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AUDIOVISUAL INFORMATION MANAGEMENT SYSTEM 

BACKGROUND OF THE INVENTION 

The present invention relates to a system for 
5 managing audiovisual information, and in particular to a 
system for audiovisual information browsing, filtering, 
searching, archiving, and personalization. 

Video cassette recorders (VCRs) may record 
video programs in response to pressing a record button or 
10 may be programmed to record video programs based on the 
time of day. However, the viewer must program the VCR 
based on information from a television guide to identify 
B relevant programs to record. After recording, the viewer 

^ scans through the entire video tape to select relevant 

ro 15 portions of the program for viewing using the 

^ functionality provided by the VCR, such as fast forward 

Ms? 

□ and fast reverse. Unfortunately, the searching and 

^ viewing is based on a linear search, which may require 

p significant time to locate the desired portions of the 

^1 20 program(s) and fast forward to the desired portion of the 

tape. In addition, it is time consuming to program the 
B VCR in light of the television guide to record desired 

programs. Also, unless the viewer recognizes the 

programs from the television guide as desirable it is 
25 unlikely that the viewer will select such programs to be 

recorded. 

RePlayTV and TiVo have developed hard disk 
based systems that receive, record, and play television 
broadcasts in a manner similar to a VCR. The systems may 

3 0 be programmed with the viewer's viewing preferences. The 
systems use a telephone line interface to receive 
scheduling information similar to that available from a 
television guide. Based upon the system programming and 
the scheduling information, the system automatically 

35 records programs that may be of potential interest to the 
viewer. Unfortunately, viewing the recorded programs 
occurs in a linear manner and may require substantial 



time. In addition, each system must be programmed for an 
individual's preference, likely in a different manner. 

Freeman et al . , U.S. Patent No. 5,861,881, 
disclose an interactive computer system where subscribers 
can receive individualized content. 

With all the aforementioned systems, each 
individual viewer is required to program the device 
according to his particular viewing preferences. 
Unfortunately, each different type of device has 
different capabilities and limitations which limit the 
selections of the viewer. In addition, each device 
includes a different interface which the viewer may be 
unfamiliar with. Further, if the operator's manual is 
inadvertently misplaced it may be difficult for the 
viewer to efficiently program the device. 

BRIEF SUMMARY OF THE INVENTION 

The present invention overcomes the 
aforementioned drawbacks of the prior art by providing a 
method of using a system, which may include, at least one 
of audio, image, and a video comprising a plurality of 
frames. A preferences description, describing 
preferences of a user with respect to the use of at least 
one of the audio, image, and video, where the description 
normally includes multiple preferences. 

The foregoing and other objectives, features 
and advantages of the invention will be more readily 
understood upon consideration of the following detailed 
description of the invention, taken in conjunction with 
the accompanying drawings . 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

FIG. 1 is an exemplary embodiment of a program, 
a system, and a user, with associated description 
schemes, of an audiovisual system of the present 
invention . 



FIG. 2 is an exemplary embodiment of the 
audiovisual system, including an analysis module, of 
FIG. 1. 

FIG. 3 is an exemplary embodiment of the 
5 analysis module of FIG. ,2. 

FIG. 4 is an illustration of a thumbnail view 
(category) for the audiovisual system. 

FIG. 5 is an illustration of a thumbnail view 
(channel) for the audiovisual system. 
10 FIG. 6 is an illustration of a text view 

(channel) for the audiovisual system. 

FIG. 7 is an illustration of a frame view for 
the audiovisual system. 

FIG. 8 is an illustration of a shot view for 
15 the audiovisual system. 

FIG. 9 is an illustration of a key frame view 
the audiovisual system. 

FIG. 10 is an illustration of a highlight view 
for the audiovisual system. 
'J^ 20 FIG. 11 is an illustration of an event view for 

p the audiovisual system. 

FIG. 12 is an illustration of a 
character/object view for the audiovisual system. 

FIG. 13 is an alternative embodiment of a 
2 5 program description scheme including a syntactic 

structure description scheme, a semantic structure 
description scheme, a visualization description scheme, 
and a meta information description scheme. 

FIG. 14 is an exemplary embodiment of the 
30 visualization description scheme of FIG. 13. 

FIG. 15 is an exemplary embodiment of the meta 
information description scheme of FIG. 13. 

FIG. 16 is an exemplary embodiment of a segment 
description scheme for the syntactic structure 
35 description scheme of FIG. 13. 
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FIG. 17 is an exemplary embodiment of a region 
description scheme for the syntactic structure 
description scheme of FIG. 13. 

FIG. 18 is an exemplary embodiment of a 
5 segment /region relation description scheme for the 
syntactic structure description scheme of FIG. 13. 

FIG. 19 is an exemplary embodiment of an event 
description scheme for the semantic structure description 
scheme of FIG. 13. 
10 FIG. 2 0 is an exemplary embodiment of an object 

description scheme for the semantic structure description 
scheme of FIG. 13. 

FIG. 21 is an exemplary embodiment of an 
event/object relation graph description scheme for the 
15 syntactic structure description scheme of FIG. 13. 

FIG- 22 is an exemplary embodiment of a user 
preference description scheme. 

FIG. 2 3 is an exemplary embodiment of the 
interrelationship between a usage history description 

2 0 scheme, an agent, and the usage preference description 

scheme of FIG. 22. 

FIG. 24 is an exemplary embodiment of the 
interrelationship between audio and/or video programs 
together with their descriptors, user identification, and 
25 the usage preference description scheme of FIG. 22. 

FIG. 2 5 is an exemplary embodiment of a usage 
preference description scheme of FIG. 22. 

FIG. 2 6 is an exemplary embodiment of the 
interrelationship between the usage description schemes 

3 0 and an MPEG-? description schemes. 

FIG. 2 7 is an exemplary embodiment of a usage 
history description scheme of FIG. 22. 

FIG, 28 is an exemplary system incorporating 
the user history description scheme. 
3 5 FIG. 2 9 is an exemplary user preferences 

description scheme . 



FIG. 30 is an exemplary user preferences 
description scheme. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Many households today have many sources of 
audio and video information, such as multiple television 
sets, multiple VCR's, a home stereo, a home entertainment 
center, cable television, satellite television, internet 
broadcasts, world wide web, data services, specialized 
Internet services, portable radio devices, and a stereo 
in each of their vehicles. For each of these devices, a 
different interface is normally used to obtain, select, 
record, and play the video and/or audio content. For 
example, a VCR permits the selection of the recording 
times but the user has to correlate the television guide 
with the desired recording times. Another example is the 
user selecting a preferred set of preselected radio 
stations for his home stereo and also presumably 
selecting the same set of preselected stations for each 
of the user's vehicles. If another household member 
desires a different set of preselected stereo selections, 
the programming of each audio device would need to be 
reprogrammed at substantial inconvenience. 

The present inventors came to the realization 
that users of visual information and listeners to audio 
information, such as for example radio, audio tapes, 
video tapes, movies, and news, desire to be entertained 
and informed in more than merely one uniform manner. In 
other words, the audiovisual information presented to a 
particular user should be in a format and include content 
suited to their particular viewing preferences. In 
addition, the format should be dependent on the content 
of the particular audiovisual information. The amount of 
information presented to a user or a listener should be 
limited to only the amount of detail desired by the 
particular user at the particular time. For example with 
the ever increasing demands on the user's time, the user 



may desire to watch only 10 minutes of or merely the 
highlights of a basketball game. In addition, the 
present inventors came to the realization that the 
necessity of programming multiple audio and visual 
devices with their particular viewing preferences is a 
burdensome task, especially when presented with 
unfamiliar recording devices when traveling. When 
traveling, users desire to easily configure unfamiliar 
devices, such as audiovisual devices in a hotel room, 
with their viewing and listening preferences in a 
efficient manner. 

The present inventors came to the further 
realization that a convenient technique of merely 
recording the desired audio and video information is not 
sufficient because the presentation of the information 
should be in a manner that is time efficient, especially 
in light of the limited time frequently available for 
the presentation of such information. In addition, the 
user should be able to access only that portion of all of 
the available information that the user is interested in, 
while skipping the remainder of the information. 

A user is not capable of watching or otherwise 
listening to the vast potential amount of information 
available through all, or even a small ' portion of, the 
sources of audio and video information. In addition, 
with the increasing information potentially available, 
the user is not likely even aware of the potential 
content of information that he may be interested in. In 
light of the vast amount of audio, image, and video 
information, the present inventors came to the 
realization that a system that records and presents to 
the user audio and video information based upon the 
user's prior viewing and listening habits, preferences, 
and personal characteristics, generally referred to as 
user information, is desirable. In addition, the system 
may present such information based on the capabilities of 
the system devices. This permits the system to record 



desirable information and to customize itself 
automatically to the user and/or listener. It is to be 
understood that user, viewer, and/or listener terms may 
be used interchangeability for any type of content. 
Also, the user information should be portable between and 
usable by different devices so that other devices may 
likewise be configured automatically to the particular 
user's preferences upon receiving the viewing 
information. 

In light of the foregoing realizations and 
motivations, the present inventors analyzed a typical 
audio and video presentation environment to determine the 
significant portions of the typical audiovisual 
environment. First, referring to FIG. 1 the video, 
image, and/or audio information 10 is provided or 
otherwise made available to a user and/or a (device) 
system- Second, the video, image, and/or audio 
information is presented to the user from the system 12 
(device), such as a television set or a radio. Third, 
the user interacts both with the system (device) 12 to 
view the information 10 in a desirable manner and has 
preferences to define which audio, image, and/or video 
information is obtained in accordance with the user 
information 14. After the proper identification of the 
different major aspects of an audiovisual system the 
present inventors then realized that information is 
needed to describe the informational content of each 
portion of the audiovisual system 16. 

With three portions of the audiovisual 
presentation system 16 identified, the functionality of 
each portion is identified together with its 
interrelationship to the other portions. To define the 
necessary interrelationships, a set of description 
schemes containing data describing each portion is 
defined. The description schemes include data that is 
auxiliary to 'the programs 10, the system 12, and the user 
14, to store a set of information, ranging from human 
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readable text to encoded data, that can be used in 
enabling browsing, filtering, searching, archiving, and 
personalization. By providing a separate description 
scheme describing the program (s) 10, the user 14, and the 
5 system 12, the three portions (program, user, and system) 
may be combined together to provide an interactivity not 
previously achievable. In addition, different programs 
10, different users 14, and different systems 12 may be 
combined together in any combination, while still 
10 maintaining full compatibility and functionality. It is 
to be understood that the description scheme may contain 
the data itself or include links to the data, as desired. 
O A program description scheme 18 related to the 

Ti video, still image, and/or audio information 10 

y I 

ffl 15 preferably includes two sets of information, namely, 

y program views and program profiles. The program views 

p define logical structures of the frames of a video that 

M define how the video frames are potentially to be viewed 

^ suitable for efficient browsing. For example the program 

U1 2 0 views may contain a set of fields that contain data for 

the identification of key frames, segment definitions 
Q between shots, highlight definitions, video summary 

O definitions, different lengths of highlights, thumbnail 

set of frames, individual shots or scenes, representative 
2 5 frame of the video, grouping of different events, and a 

close-up view. The program view descriptions may contain 
thumbnail, slide, key frame, highlights, and close-up 
views so that users can filter and search not only at the 
program level but also within a particular program. The 
30 description scheme also enables users to access 

information in varying detail amounts by supporting, for 
example, a key frame view as a part of a program view 
providing multiple levels of summary ranging from coarse 
to fine. The program profiles define distinctive 
35 characteristics of the content of the program, such as 
actors, stars, rating, director, release date, time 
stamps, keyword identification, trigger profile, still 
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profile, event profile, character profile, object 
profile, color profile, texture profile, shape profile, 
motion profile, and categories. The program profiles are 
especially suitable to facilitate filtering and searching 
of the audio and video information. The description 
scheme enables users to have the provision of discovering 
interesting programs that they may be unaware of by 
providing a user description scheme. The user 
description scheme provides information to a software 
agent that in turn performs a search and filtering on 
behalf of the user by possibly using the system 
description scheme and the program description scheme 
information. It is to be understood that in one of the 
embodiments of the invention merely the program 
description scheme is included. 

Program views contained in the program 
description scheme are a feature that supports a 
functionality such as close-up view. In the close-up 
view, a certain image object, e.g., a famous basketball 
player such as Michael Jordan, can be viewed up close by 
playing back a close-up sequence that is separate from 
the original program. An alternative view can be 
incorporated in a straightforward manner. Character 
profile on the other hand may contain spatio-temporal 
position and size of a rectangular region around the 
character of interest. This region can be enlarged by 
the presentation engine, or the presentation engine may 
darken outside the region to focus the user * s attention 
to the characters spanning a certain number of frames. 
Information within the program description scheme may 
contain data about the initial size or location of the 
region, movement of the region from one frame to another, 
and duration and terms of the number of frames featuring 
the region. The character profile also provides 
provision for including text annotation and audio 
annotation about the character as well as web page 
information, and any other suitable information. Such . 
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character profiles may include the audio annotation which 
is separate from and in addition to the associated audio 
track of the video. 

The program description scheme may likewise 
5 contain similar information regarding audio (such as 

radio broadcasts) and images (such as analog or digital 
photographs or a frame of a video) . 

The user description scheme 2 0 preferably 
includes the user * s personal preferences, and information 
10 regarding the user's viewing history such as for example 
browsing history, filtering history, searching history, 
and device setting history. The user's personal 
Q preferences includes information regarding particular 

}5 programs and categorizations of programs that the user 

15 prefers to view. The user description scheme may also 
include personal information about the particular user, 
5 such as demographic and geographic information, e.g. zip 

63 code and age. The explicit definition of the particular 

L. programs or attributes related thereto permits the system 

yi 2 0 16 to select those programs from the information 

^ contained within the available program description 

y ' 

Q schemes 18 that may be of interest to the user. 

O Frequently, the user does not desire to learn to program 

the device nor desire to explicitly program the device. 
25 In addition, the user description scheme 2 0 may not be 
sufficiently robust to include explicit definitions 
describing all desirable programs for a particular user. 
In such a case, the capability of the user description 
scheme 2 0 to adapt to the viewing habits of the user to 
30 accommodate different viewing characteristics not 
explicitly provided for or otherwise difficult to 
describe is useful. In such a case, the user description 
scheme 2 0 may be augmented or any technique can be used 
to compare the information contained in the user 
3 5 description scheme 2 0 to the available information 

contained in the program description scheme 18 to make 
selections. The user description scheme provides a 



technique for holding user preferences ranging from 
program categories to program views, as well as usage 
history. User description scheme information is 
persistent but can be updated by the user or by an 
intelligent software agent on behalf of the user at any 
arbitrary time. It may also be disabled by the user, at 
any time, if the user decides to do so. In addition, the 
user description scheme is modular and portable so that 
users can carry or port it from one device to another, 
such as with a handheld electronic device or smart card 
or transported over a network connecting multiple 
devices. When user description scheme is standardized 
among different manufacturers or products, user 
preferences become portable. For example, a user can 
personalize the television receiver in a hotel room 
permitting users to access information they prefer at any 
time and anywhere. In a sense, the user description 
scheme is persistent and timeless based. In addition, 
selected information within the program description 
scheme may be encrypted since at least part of the 
information may be deemed to be private (e.g., 
demographics) . A user description scheme may be 
associated with an audiovisual program broadcast and 
compared with a particular user's description scheme of 
the receiver to readily determine whether or not the 
program's intended audience profile matches that of the 
user. It is to be understood that in one of the 
embodiments of the invention merely the user description 
scheme is included. 

The system description scheme 22 preferably 
manages the individual programs and other data. The 
management may include maintaining lists of programs, 
categories, channels, users, videos, audio, and images. 
The management may include the capabilities of a device 
for providing the audio, video, and/or images. Such 
capabilities may include, for example, screen size, 
stereo, AC3 , DTS, color, black/white, etc. The 
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management may also include relationships between any one 
or more of the user, the audio, and the images in 
relation to one or more of a program description 
scheme (s) and a user description scheme (s) . In a similar 
manner the management may include relationships between 
one or more of the program description scheme (s) and user 
description scheme (s). It is to be understood that in 
one of the embodiments of the invention merely the system 
description scheme is included. 

The descriptors of the program description 
scheme and the user description scheme should overlap, at 
least partially, so that potential desirability of the 
program can be determined by comparing descriptors 
representative of the same information. For example, the 
program and user description scheme may include the same 
set of categories and actors. The program description 
scheme has no knowledge of the user description scheme, 
and vice versa, so that each description scheme is not 
dependant on the other for its existence. It is not 
necessary for the description schemes to be fully 
populated. It is also beneficial not to include the 
program description scheme with the user description 
scheme because there will likely be thousands of programs 
with associated description schemes which if combined 
with the user description scheme would result in a 
unnecessarily large user description scheme. It is 
desirable to maintain the user description scheme small 
so that it is more readily portable. Accordingly, a 
system including only the program description scheme and 
the user description scheme would be beneficial . 

The user description scheme and the system 
description scheme should include at least partially 
overlapping fields. With overlapping fields the system 
can capture the desired information, which would 
otherwise not be recognized as desirable. The system 
description scheme preferably includes a list of users 
and available programs. Based on the master list of 
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available programs, and associated program description 
scheme, the system can match the desired programs. It is 
also beneficial not to include the system description 
scheme with the user description scheme because there 
5 will likely be thousands of programs stored in the system 
description schemes which if combined with the user 
description scheme would result in a unnecessarily large 
user description scheme. It is desirable to maintain 

the user description scheme small so that it is more 
10 readily portable. For example, the user description 

scheme may include radio station preselected frequencies 
and/or types of stations, while the system description 
p scheme includes the available stations for radio stations 

in particular cities. When traveling to a different city 

Ln 

m 15 the user description scheme together with the system 

description scheme will permit reprogramming the radio 
stations. Accordingly, a system including only the 
K system description scheme and the user description scheme 

would be beneficial, 
[n 2 0 The program description scheme and the system 

ry description scheme should include at least partially 

overlapping fields. With the overlapping fields, the 
system description scheme will be capable of storing the 
information contained within the program description 

2 5 scheme, so that the information is properly indexed. 
With proper indexing, the system is capable of matching 
such information with the user information, if available, 
for obtaining and recording suitable programs. If the 
program description scheme and the system description 

3 0 scheme were not overlapping then no information would be 
extracted from the programs and stored. System 
capabilities specified within the system description 
scheme of a particular viewing system can be correlated 
with a program description scheme to determine the views 

3 5 that can be supported by the viewing system. For 

instance, if the viewing device is not capable of playing 
back video, its system description scheme may describe 
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its viewing capabilities as limited to keyframe view and 
slide view only. Program description scheme of a 
particular program and system description scheme of the 
viewing system are utilized to present the appropriate 
views to the viewing system. Thus, a server of programs 
serves the appropriate views according to a particular 
viewing system's capabilities, which may be communicated 
over a network or communication channel connecting the 
server with user's viewing device. It is preferred to 
maintain the program description scheme separate from the 
system description scheme because the content providers 
repackage the content and description schemes in 
different styles, times, and formats. Preferably, the 
program description scheme is associated with the 
program, even if displayed at a different time. 
Accordingly, a system including " only the system 
description scheme and the program description scheme 
would be beneficial. 

By preferably maintaining the independence of 
each of the three description schemes while having fields 
that correlate the same, information, the programs 10, the 
users 14, and the system 12 may be interchanged with one 
another while maintaining the functionality of the entire 
system 16. Referring to FIG. 2, the audio, visual, or 
audiovisual program 38, is received by the system 16. 
The program 3 8 may originate at any suitable source, such 
as for example broadcast television, cable television, 
satellite television, digital television, Internet 
broadcasts, world wide web, digital video discs, still 
images, video cameras, laser discs, magnetic media, 
computer hard drive, video tape, audio tape, data 
services, radio broadcasts, and microwave communications. 
The program description stream may originate from any 
suitable source, such as for example PSIP/DVB-SI 
information in digital television broadcasts, specialized 
digital television data services, specialized Internet 
services, world wide web, data files, data over the 
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telephone, and memory, such as computer memory. The 
program, user, and/or system description scheme may be 
transported over a network (communication channel) . For 
example, the system description scheme may be transported 
5 to the source to provide the source with views or other 
capabilities that the device is capable of using. In 
response, the source provides the device with image, 
audio, and/or video content customized or otherwise 
suitable for the particular device. The system 16 may 
10 include any device (s) suitable to receive any one or more 
of such programs 38. An audiovisual program analysis 
module 42 performs an analysis of the received programs 
Q 3 8 to extract and provide program related information 

'f£ (descriptors) to the description scheme (DS) generation 

Li : 

m 15 module 44 . The program related information may be 

^ extracted from the data stream including the program 3 8 

03 

g or obtained from any other source, such as for example 

W data transferred over a telephone line, data already 

L.^ transferred to the system 16 in the past, or data from an 

Ul 20 associated file. The program related information 

iM preferably includes data defining both the program views 

p and the program profiles available for the particular 

O program 38. The analysis module 42 performs an analysis 

of the programs 3 8 using information obtained from (i) 
25 automatic audio-video analysis methods on the basis of 

low- level features that are extracted from the. 

program (s) , (ii) event detection techniques, (iii) data 

that is available (or extractable) from data sources or 

electronic program guides (EPGs, DVB-SI, and PSIP) , and 
3 0 (iv) user information obtained from the user description 

scheme 2 0 to provide data defining the program 

descript ion scheme . 

The selection of a particular program analysis 

technique depends on the amount of readily available data 
3 5 and the user preferences. For example, if a user prefers 

to watch a 5 minute video highlight of a particular 

program, such as a basketball game, the analysis module 
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4 2 may invoke a knowledge based system 90 (FIG. 3) to 
determine the highlights that form the best 5 minute 
summary. The knowledge based system 90 may invoke a 
commercial filter 92 to remove commercials and a slow 
5 motion detector 54 to assist in creating the video 

summary. The analysis module 42 may also invoke other 
modules to bring information together (e.g., textual 
information) to author particular program views. For 
example, if the program 3 8 is a home video where there is 
10 no further information available then the analysis module 
42 may create a key- frame summary by identifying key- 
frames of a multi- level summary and passing the 
information to be used to generate the program views, and 
in particular a key frame view, to the description 
15 scheme. Referring also to FIG. 3, the analysis module 42 
may also include other sub-modules, such as for example, 
a de-mux/decoder 60, a data and service content analyzer 
62, a text processing and text summary generator 64, a 
% close caption analyzer 66, a title frame generator 68, an 

Ln 20 analysis manager 70, an audiovisual analysis and feature 

^ extractor 72, an event detector 74, a key- frame 

PI summarizer 76, and a highlight summarizer 78. * 

Q The generation module 44 receives the system 

information 46 for the system description scheme. The 

2 5 system information 46 preferably includes data for the 

system description scheme 22 generated by the generation 
module 44. The generation module 44 also receives user 
information 4 8 including data for the user description 
scheme. The user information 48 preferably includes data 

3 0 for. the user description scheme generated within the 

generation module 44. The user input 48 may include, for 
example, meta information to be included in the program 
and system description scheme. The user description 
scheme (or corresponding information) is provided to the 
35 analysis module 42 for selective analysis of the 

program(s) 38. For example, the user description scheme 
may be suitable for triggering the highlight generation 



functionality for a particular program and thus 
generating the preferred views and storing associated 
data in the program description scheme. The generation 
module 44 and the analysis module 42 provide data to a 
data storage unit 50. The storage unit 50 may be any 
storage device, such as memory or magnetic media. 

A search, filtering, and browsing (SFB) module 
52 implements the description scheme technique by parsing 
and extracting information contained within the 
description scheme. The SFB module 52 may perform 
filtering, searching, and browsing of the programs 38, on 
the basis of the information contained in the description 
schemes. An intelligent software agent is preferably 
included within the SFB module 52 that gathers and 
provides user specific information to the generation 
module 44 to be used in authoring and updating the user 
description scheme (through the generation module 44) . 
In this manner, desirable content may be provided to the 
user though a display 80. The selections of the desired 
program (s) to be retrieved, stored, and/or viewed may be 
programmed, at least in part, through a graphical user 
interface 82. The graphical user interface may also 
include or be connected to a presentation engine for 
presenting the information to the user through the 
graphical user interface . 

The intelligent management and consumption of 
audiovisual information using the mult i -part description 
stream device provides a next -generation device suitable 
for the modern era of information overload. The device 
responds to changing lifestyles of individuals and 
families, and allows everyone to obtain the information 
they desire anytime and anywhere they want. 

An example of the use of the device may be as 
follows. A user comes home from work late Friday evening 
being happy the work week is finally over. The user 
desires to catch up with the events of the world and then 
watch ABC's 2 0/2 0 show later that evening. It is now 9 
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PM and the 20/20 show will start in an hour at 10 PM. 
The user is interested in the sporting events of the 
week, and all the news about the Microsoft case with the 
Department of Justice. The user description scheme may- 
include a profile indicating a desire that the particular 
user wants to obtain all available information regarding 
the Microsoft trial and selected sporting events for 
particular teams. In addition, the system description 
scheme and program description scheme provide information 
regarding the content of the available information that 
may selectively be obtained and recorded. The system, in 
an autonomous manner, periodically obtains and records 
the audiovisual information that may be of interest to 
the user during the past week based on the three 
description schemes. The device most likely has recorded 
more than one hour of audiovisual information so the 
information needs to be condensed in some manner. The 
user starts interacting with the system with a pointer or 
voice commands to indicate a desire to view recorded 
sporting programs. On the display, the user is presented 
with a list of recorded sporting events including 
Basketball and Soccer. Apparently the user's favorite 
Football team did not play that week because it was not 
recorded. The user is interested in basketball games and 
indicates a desire to view games. A set of title frames 
is presented on the display that captures an important 
moment of each game. The user selects the Chicago Bulls 
game and indicates a desire to view a 5 minute highlight 
of the game. The system automatically generates 
highlights. The highlights may be generated by audio or 
video analysis, or the program description scheme 
includes data indicating the frames that are presented 
for a 5 minute highlight. The system may have also 
recorded web-based textual information regarding the 
particular Chicago-Bulls game which may be selected by 
the user for viewing. If desired, the summarized 
information may be recorded onto a storage device, such 
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as a DVD with a label. The stored information may also 
include an index code so that it can be located at a 
later time. After viewing the sporting events the user 
may decide to read the news about the Microsoft trial . 
5 It is now 9:50 PM and the user is done viewing the news. 
In fact, the user has selected to delete all the recorded 
news items after viewing them. The user then remembers 
to do one last thing before 10 PM in the evening. The 
next day, the user desires to watch the VHS tape that he 
10 received from his brother that day, containing footage 
about his brother's new baby girl and his vacation to 
Peru last summer. The user wants to watch the whole 2- 
D hour tape but he is anxious to see what the baby looks 

yd like and also the new stadium built in Lima, which was 

: i : 
^^' t 

m 15 not there last time he visited Peru. The user plans to 
y take a quick look at a visual summary of the tape, 

g browse, and perhaps watch a few segments for a couple of 

S minutes, before the user takes his daughter to her piano 

s 

Q lesson at 10 AM the next morning. The user plugs in the 

y\ 20 tape into his VCR, that is connected to the system, and 

invokes the summarization functionality of the system to 
□ scan the tape and prepare a summary. The user can then 

O view the summary the next morning to quickly discover the 

baby's looks, and playback segments between the key- 

2 5 frames of the summary to catch a glimpse of the crying 

baby. The system may also record the tape content onto 
the system hard drive (or storage device) so the video 
summary can be viewed quickly. It is now 10:10 PM, and 
it seems that the user is 10 minutes late for viewing 
30 20/20. Fortunately, the system, based on the three 

description schemes, has already been recording 2 0/2 0 
since 10 PM. Now the user can start watching the 
recorded portion of 20/20 as the recording of 20/20 
proceeds. The user will be done viewing 20/20 at 11:10 

3 5 PM. 

The average consumer has an ever increasing 
number of multimedia devices, such as a home audio 
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system, a car stereo, several home television sets, web 
browsers, etc. The user currently has to customize each 
of the devices for optimal viewing and/or listening 
preferences. By storing the user preferences on a 
5 removable storage device, such as a smart card, the user 
may insert the card including the user preferences into 
such media devices for automatic customization. This 
results in the desired programs being automatically 
recorded on the VCR, and setting of the radio stations 
10 for the car stereo and home audio system. In this manner 
the user only has to specify his preferences at most 
once, on a single device and subsequently, the 
□ descriptors are automatically uploaded into devices by 

yd the removable storage device. The user description 

^ 15 scheme may also be loaded into other devices using a 

y wired or wireless network connection, e.g. that of a home 

^ network. Alternatively, the system can store the user 

03 history and create entries in the user description scheme 

f~ based on the ' s audio and video viewing habits. In this 

yl 2 0 manner, the user would never need to program the viewing 

IM information to obtain desired information. In a sense, 

- 

P the user descriptor scheme enables modeling of the user 

^ by providing a central storage for the user * s listening, 

viewing, browsing preferences, and user's behavior. This 
25 enables devices to be quickly personalized, and enables 
other components, such as intelligent agents, to 
communicate on the basis of a standardized description 
format, and to make smart inferences regarding the user's 
preferences. 

3 0 Many different realizations and applications 

can be readily derived from FIGS. 2 and 3 by 
appropriately organizing and utilizing their different 
parts, or by adding peripherals and extensions as needed. 
In its most general form, FIG. 2 depicts an audiovisual 

35 searching, filtering, browsing, and/or recording 

appliance that is personalizable . The list of more 



specific applications/implementations given below is not 
exhaustive but covers a range. 

The user description scheme is a major enabler 
for personalizable audiovisual appliances. If the 
structure (syntax and semantics) of the description 
schemes is known amongst multiple appliances , the user 
(user) can carry (or otherwise transfer) the information 
contained within his user description scheme from one 
appliance to another, perhaps via a smart card- -where 
these appliances support smart card interface-- in order 
to personalize them. Personalization can range from 
device settings, such as display contrast and volume 
control, to settings of television channels, radio 
stations, web stations, web sites, geographic 
information, and demographic information such as age, zip 
code etc. Appliances that can be personalized may access 
content from different sources. They may be connected to 
the web, terrestrial or cable broadcast, etc., and they 
may also access multiple or different types of single 
media such as video, music, etc. 

For example, one can personalize the car stereo 
using a smart card plugged out of the home system and 
plugged into the car stereo system to be able to tune to 
favorite stations at certain times. As another example, 
one can also personalize television viewing, for example, 
by plugging the smart card into a remote control that in 
turn will autonomously command the television receiving 
system to present the user information about current and 
future programs that fits the user's preferences. 
Different members of the household can instantly 
personalize the viewing experience by inserting their own 
smart card into the family remote. In the absence of such 
a remote, this same type of personalization can be 
achieved by plugging in the smart card directly to the 
television system. The remote may likewise control audio 
systems. In another implementation, the television 
receiving system holds user description schemes for 
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multiple users (users) in local storage and identify 
different users (or group of users) by using an 
appropriate input interface. For example an interface 
using user-voice identification technology. It is noted 
5 that in a networked system the user description scheme 
may be transported over the network. 

The user description scheme is generated by 
direct user input, and by using a software that watches 
the user to determine his/her usage pattern and usage 
10 history. User description scheme can be updated in a 
dynamic fashion by the user or automatically. A well 
defined and structured description scheme design allows 
different devices to interoperate with each other. A 
modular design also provides portability. 
15 The description scheme adds new functionality 

Q to those of the current VCR. An advanced VCR system can 

Q learn from the user via direct input of preferences, or 

K by watching the usage pattern and history of the user. 

1:. The user description scheme holds user's preferences 

Ln 2 0 users and usage history. An intelligent agent can then 

consult with the user description scheme and obtain 
information that it needs for acting on behalf of the 
user. Through the intelligent agent, the system acts on 
behalf of the user to discover programs that fit the 
25 taste of the user, alert the user about such programs, 

and/or record them autonomously. An agent can also manage 
the storage in the system according to the user 
description scheme, i.e., prioritizing the deletion of 
programs (or alerting the user for transfer to a 
3 0 removable media) , or determining their compression factor 
(which directly impacts their visual quality) according 
to user's preferences and history. 

The program description scheme and the system 
description scheme work in collaboration with the user 
35 description scheme in achieving some tasks. In addition, 
the program description scheme and system description 
scheme in an advanced VCR or other system will enable the 
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user to browse, search, and filter audiovisual programs. 
Browsing in the system offers capabilities that are well 
beyond fast forwarding and rewinding. For instance, the 
user can view a thumbnail view of different categories of 
5 programs stored in the system. The user then may choose 
frame view, shot view, key frame view, or highlight view, 
depending on their availability and user's preference. 
These views can be readily invoked using the relevant 
information in the program description scheme, especially 

10 in program views. The user at any time can start viewing 
the program either in parts, or in its entirety. 

In this application, the program description 
scheme may be readily available from many services such 
S as: (i) from broadcast (carried by EPG defined as a part 

15 of ATSC-PSIP (ATSC-Program Service Integration Protocol) 

yy 

Q in USA or DVB-SI (Digital Video Broadcast-Service 

^ Information) in Europe) ; (ii) from specialized data 

m services (in addition to PSIP/DVB-SI) ; (iii) from 

^ specialized web sites; (iv) from the media storage unit 

p 20 containing the audiovisual content (e.g., DVD); (v) from 

ru advanced cameras (discussed later) , and/or may be 

yj generated (i.e., for programs that are being stored) by 

B the analysis module 42 or by user input 48. 

Contents of digital still and video cameras can 
2 5 be stored and managed by a system that implements the 

description schemes, e.g., a system as shown in FIG. 2. 
Advanced cameras can store a program description scheme, 
for instance, in addition to the audiovisual content 
itself. The program description scheme can be generated 
30 either in part or in its entirety on the camera itself 
via an appropriate user input interface (e.g., speech, 
visual menu drive, etc.). Users can input to the camera 
the program description scheme information, especially 
those high-level (or semantic) information that may 
35 otherwise be difficult to automatically extract by the 

system. Some camera settings and parameters (e.g., date 
and time) , as well as quantities computed in the camera 



(e.g., color histogram to be included in the color 
profile) , can also be used in generating the program 
description scheme. Once the camera is connected, the 
system can browse the camera content, or transfer the 
camera content and its description scheme to the local 
storage for future use. It is also possible to update or 
add information to the description scheme generated in 
the camera . 

The IEEE 13 94 and Havi standard specifications 
enable this type of "audiovisual content" centric 
communication among devices. The description scheme 
API's can be used in the context of Havi to browse and/or 
search the contents of a camera or a DVD which also 
contain a description scheme associated with their 
content, i.e., doing more than merely invoking the PLAY 
API to play back and linearly view the media. 

The description schemes may be used in 
archiving audiovisual programs in a 
database. The search engine uses the information 
contained in the program description scheme to retrieve 
programs on the basis of their content. The program 
description scheme can also 

be used in navigating through the contents of the 
database or the query results. The user description 
scheme can be used in prioritizing the results of the 
user query during presentation. It is possible of course 
to make the program description scheme more comprehensive 
depending on the nature of the particular application. 

The description scheme fulfills the user's 
desire to have applications that pay attention and are 
responsive to their viewing and usage habits, 
preferences, and personal demographics. The proposed 
user description scheme directly addresses this desire in 
its selection of fields and interrelationship to other 
description schemes. Because the description schemes are 
modular in nature, the user can port his user description 



scheme from one device to another in order to 
"personalize" the device. 

The proposed description schemes can be 
incorporated into current products similar to those from 
TiVo and Replay TV in order to extend their entertainment 
informational value. In particular, the description 
scheme will enable audiovisual browsing and searching of 
programs and enable filtering within a particular program 
by supporting multiple program views such as the 
highlight view. In addition, the description scheme will 
handle programs coming from sources other than television 
broadcasts for which TiVo and Replay TV are not designed 
to handle. In addition, by standardization of TiVo and 
Replay TV type of devices, other products may be 
interconnected to such devices to extend their 
capabilities, such as devices supporting an MPEG 7 
description. MPEG- 7 is the Moving Pictures Experts Group 
- 7, acting to standardize descriptions and description 
schemes for audiovisual information. The device may also 
be extended to be personalized by multiple users, as 
desired . 

Because the description scheme is defined, the 
intelligent software agents can communicate among 
themselves to make intelligent inferences regarding the 
user's preferences. In addition, the development and 
upgrade of intelligent software agents for browsing and 
filtering applications can be simplified based on the 
standardized user description scheme. 

The description scheme is multi -modal in the 
following sense that it holds both high level (semantic) 
and low level features and/or descriptors. For example, 
the high and low level descriptors are actor name and 
motion model parameters, respectively. High level 
descriptors are easily readable by humans while low level 
descriptors are more easily read by machines and less 
understandable by humans. The program description scheme 
can be readily harmonized with existing EPG, PSIP, and 



DVB-SI information facilitating search and filtering of 
broadcast programs. Existing services can be extended in 
the future by incorporating additional information using 
the compliant description scheme. 

For example, one case may include audiovisual 
programs that are prerecorded on a media such as a 
digital video disc where the digital video disc also 
contains a description scheme that has the same syntax 
and semantics of the description scheme that the FSB 
module uses. If the FSB module uses a different 
description scheme, a transcoder (converter) of the 
description scheme may be employed. The user may want to 
browse and view the content of the digital video disc. 
In this case, the user may not need to invoke the 
analysis module to author a program description. 
However, the user may want to invoke his or her user 
description scheme in filtering, searching and browsing 
the digital video disc content. Other sources of program 
information may likewise be used in the same manner. 

It is to be understood that any of the 
techniques described herein with relation to video are 
equally applicable to images (such as still image or a 
frame of a video) and audio (such as radio) . 

An example of an audiovisual interface is shown 
in FIGS. 4-12 which is suitable for the preferred 
audiovisual description scheme. Referring to FIG. 4, by 
selecting the thumbnail function as a function of 
category provides a display with a set of categories on 
the left hand side. Selecting a particular category, 
such as news, provides a set of thumbnail views of 
different programs that are currently available for 
viewing. In addition, the different programs may also 
include programs that will be available at a different 
time for viewing. The thumbnail views are short video 
segments that provide an indication of the content of the 
respective actual program that it corresponds with. 
Referring to FIG. 5, a thumbnail view of available 



programs in terms of channels may be displayed, if 
desired. Referring to FIG. 6, a text view of available 
programs in terms of channels may be displayed, if 
desired. Referring to FIG. 7, a frame view of particular 
programs may be displayed, if desired. A representative 
frame is displayed in the center of the display with a 
set of representative frames of different programs in the 
left hand column. The frequency of the number of frames 
may be selected, as desired. Also a set of frames are 
displayed on the lower portion of the display 
representative of different frames during the particular 
selected program. Referring to FIG. 8, a shot view of 
particular programs may be displayed, as desired. A 
representative frame of a shot is displayed in the center 
of the display with a set of representative frames of 
different programs in the left hand column. Also a set 
of shots are displayed on the lower portion of the 
display representative of different shots (segments of a 
program, typically sequential in nature) during the 
particular selected program. Referring to FIG. 9, a key 
frame view of particular programs may be displayed, as 
desired- A representative frame is displayed in the 
center of the display with a set of representative frames 
of different programs in the left hand column. Also a 
set of key frame views are displayed on the lower portion 
of the display representative of different key frame 
portions during the particular selected program. The 
number of key frames in each key frame view can be 
adjusted by selecting the level. Referring to FIG. 10, a 
highlight view may likewise be displayed, as desired. 
Referring to FIG. 11, an event view may likewise be 
displayed, as desired. Referring to FIG. 12, a 
character/obj ect view may likewise be displayed, as 
desired . 

An example of the description schemes is shown 
below in XML. The description scheme may be implemented 
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in any language and include any of the included 
descriptions (or more) , as desired. 

The proposed program description scheme 
includes three major sections for describing a video 
5 program. The first section identifies the described 

program. The second section defines a number of views 
which may be useful in browsing applications. The third 
section defines a number of profiles which may be useful 
in filtering and search applications. Therefore, the 
10 overall structure of the proposed description scheme is 
as follows: 

<?XML version="1.0"> 

<!DOCTYPE MPEG-7 SYSTEM "mpeg-7 . dtd"> 

<PrograinIdentity> 

15 <PrograinID> . . . </PrograinID> 

<ProgramNaTne> . . . </ProgramNaine> 

<SourceLocation> . . . </SourceLocation> 
</ProgramIdentity> 
<PrograitiViews> 

2 0 <ThiainbnailView> . . . </ThumbnailView> 
<SlideView> . . . </SlideView> 
<FraineView> . , . </FrameView> 
<ShotView> . . . </ShotView> 
<KeyFrameView> . . . </KeyFrameView> 

2 5 <HighlightView> . . . </HighlightView> 

<EventView> . . . </EventView> 

<CloseUpView> . . . </CloseUpView> 

<AlternateView> . . . </AlternateView> 
</ProgramViews> 
BEQtogramProf iles> 

<GeneralProf ile> . . . </GeneralProf ile> 

<CategoryProf ile> . . . </CategoryProf ile> 

<DateTimeProf ile> . . . </DateTimeProf ile> 

<KeywordProf ile> . . . </KeywordProf ile> 

3 5 <TriggerProf ile> . . . </TriggerProf ile> 

<StillProfile> . . . </StillProf ile> 
<EventProf ile> . . . </EventProf ile> 
<CharacterProf ile> . . . </CharacterProf ile> 
<ObjectProf ile> . . . </ObjectProf ile> 

4 0 <ColorProf ile> . . . </ColorProf ile> 

<TextureProf ile> . . . </TextureProf ile> 
<ShapeProf ile> , . . </ShapeProf ile> 
<MotionProf ile> . . . </MotionProf ile> 
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</PrograinProf iles> 

Program Identity 

• Program ID 

5 

<ProgramID> program-id </PrograniID> 

The descriptor <ProgramID> contains a number or a string 
to identify a program. 

10 • Program name 

<PrograinName> program-name </ProgramName> 

The descriptor <ProgramName> specifies the name of a 
15 program. 

• Source location 

<SourceLocation> source-url </SourceLocation> 

2 0 The descriptor <SourceLocation> specifies the location of 

a program in URL format . 

Program Views 

• Thumbnail view 

2 5 <ThiimbnailView> 

<Image> thumbnail- image </Image> 
</ThumbnailView> 

The descriptor <ThumbnailView> specifies an image as the 

3 0 thumbnail representation of a program. 
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• Slide view 

<SlideView> frame-id . . . </SlideView> 

5 The descriptor <SlideView> specifies a number of frames 

in a program which may be viewed as snapshots or in a 
slide show manner. 

• Frame view 

1 0 <FrameView> start-frame-id end-frame-id </FrameView> 

The descriptor <FrameView> specifies the start and end 
frames of a program. This is the most basic view of a 
program and any program has a frame view. 

15 • Shot view 
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<ShotView> 

<Shot id=""> start-frame-id end-frame-id display-frame-id </Shot> 
yj <Shot id=""> start-frame-id end-frame-id display-frame-id </Shot> 

20 

</ShotView> 

The descriptor <ShotView> specifies a number of shots in 
a program. The <Shot> descriptor defines the start and 
25 end frames of a shot. It may also specify a frame to 

represent the shot . 



• Key- frame view 



<KeyFrameView> 
3 0 <KeyFrames level=""> 

<Clip id=""> start-frame-id end-frame-id display-f rarae-id </Clip> 
<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</KeyFrames> 
3 5 <KeyFrames level=""> 
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<Clip id=""> St art -frame -id end-f rame-id display-frame-id </Clip> 
<Clip id-""> start-frame-id end-frame-id display-frame-id </Clip> 

</KeyFrames> 

5 

</KeyFrameView> 

The descriptor <KeyFrameView> specifies key frames in a 
program. The key frames may be organized in a 
10 hierarchical manner and the hierarchy is captured by the 

descriptor <KeyFrames> with a level attribute. The clips 
which are associated with each key frame are defined by 
the descriptor <Clip>. Here the display frame in each 
clip is the corresponding key frame. 

15 • Highlight view 



<HighlightView> 

<Highlight length=""> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 
2 0 <Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Highlight> 
<Highlight length=""> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

2 5 <Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Highlight> 

</HighlightView> 

30 

The descriptor <HighlightView> specifies .clips to form 
highlights of a program. A program may have different 
versions of highlights which are tailored into various 
time length. The clips are grouped into each version of 
35 highlight which is specified by the descriptor 

<Highlight> with a length attribute. 



• Event view 



<EventView> 

<Events name=""> 

5 <Clip id=""> start- frame-id end-frame-id display-frame-id </Clip> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Events> 
<Events name=""> 

10 <Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Events> 

15 </EventView> 

The descriptor <EventView> specifies clips which are 
related to certain events in a program. The clips are 
grouped into the corresponding events which are specified 
20 by the descriptor <Event> with a name attribute. 

• Close-up view 



<CloseUpView> 

<Target name=""> 

2 5 <Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Target> 
<Target name=""> 

3 0 <Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

<Clip id=""> start-frame-id end-frame-id display-frame-id </Clip> 

</Target> 
3 5 </CloseUpView> 

The descriptor <CloseUpView> specifies clips which may be 
zoomed in to certain targets in a program. The clips are 
grouped into the corresponding targets which are 
40 specified by the descriptor <Target> with a name 

attribute . 
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Alternate view 



<AlternateView> 

<AlternateSource id=""> source-url </AlternateSource> 
5 <AlternateSource id=""> source-url </AlternateSource> 



</AlternateView> 

The descriptor <AlternateView> specifies sources which 
10 may be shown as alternate views of a program. Each 

alternate view is specified by the descriptor 
<AlternateSource> with an id attribute. The locate of the 
source may be specified in URL format . 

Program Profiles 
15 • General profile 



fz=i <GeneralProf ile> 

r^. <Title> title-text </Title> 

nj <Abstract> abstract-text </Abstract> 

pi 2 0 <Audio> voice-annotation </Audio> 

Q <Www> web-page-url </Www> 

1=1 <ClosedCaption> yes/no </ClosedCaption> 

<Language> language-name </Language> 

<Rating> rating </Rating> 
2 5 <Length> time </Length> 

<Authors> author-name . . . </Authors> 

<Producers> producer-name . . . </Producers> 

<Directors> director-name . . . </Directors> 

<Actors> actor-name . . . </Actors> 

30 

</GeneralProf ile> 

The descriptor <GeneralProf ile> describes the general 
aspects of a program. 



3 5 • Category profile 



<CategoryProf ile> category-name . . . </CategoryProf ile> 



The descriptor <CategoryProf ile> specifies the categories 
under which a program may be classified. 



• Date- time profile 

5 

<DateTimeProf ile> 

<ProductionDate> date </ProductionDate> 
<ReleaseDate> date </ReleaseDate> 
<RecordingDate> date </RecordingDate> 
0 <RecordingTime> time </RecordingTime> 

</DateTimeProf ile> 

The descriptor <DateTimeProf ile> specifies various date 
5 and time information of a program. 

• Keyword profile 



<KeywordProf ile> keyword . . . </KeywordProf ile> 

0 The descriptor <KeywordProf ile> specifies a number of 

keywords which may be used to filter or search a program. 



• Trigger profile 

<TriggerProf ile> trigger-f rame-id . . . </TriggerProf ile> 

5 

The descriptor <TriggerProf ile> specifies a number of 
frames in a program which may be used to trigger certain 
actions while the playback of the program. 



• Still profile 

0 

<StillProf ile> 

<Still id=""> 

<HotRegion id =""> 
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<Location> xl yl x2 y2 </Location> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 
</HotRegion> 
<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 
</HotRegion> 

</Still> 
<Still id=""> 

<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 
</HotRegion> 
<HotRegion id =""> 

<Location> xl yl x2 y2 </Location> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
<Www> web-page-url </Www> 
</HotRegion> 

</Still> 

30 </StillProfile> 

. The descriptor <StillProf ile> specifies hot regions or 
regions of interest within a frame. The frame is 
specified by the descriptor <Still> with an id attribute 

3 5 which corresponds to the frame -id. Within a frame, each 

hot region is specified by the descriptor <HotRegion> 
with an id attribute. 

• Event profile 

4 0 <EventProfile> 

<EventList> event-name . . . </EventList> 
< Event naine=""> 
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<Www> web-page-url </Www> 
<Occurrence id=""> 

<Duration> start -frame- id end-f raitie-id </Duration> 

<Text> text-annotation </Text> 
5 <Audio> voice-annotation </Audio> 

</Occurrence> 
<Occurrence id=""> 

<Duration> start-f rame-id end-frame-id </Duration> 

<Text> text-annotation </Text> 
0 <Audio> voice-annotation </Audio> 

</Occurrence> 

</Event> 
<Event name=""> 
5 <Www> web-page-url </Www> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
0 </Occurrence> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
5 </Occurrence> 

</Event> 

< /Event Prof ile> 

0 

The descriptor <EventProf ile> specifies the detailed 
information for certain events in a program. Each event 
is specified by the descriptor <Event> with a name 
attribute. Each occurrence of an event is specified by 
5 the descriptor <Occurrence> with an id attribute which 

may be matched with a clip id under <Ev.entView> . 



• Character profile 



<CharacterProf ile> 
0 <CharacterList> character-name . . . </CharacterList> 

<Character name=""> 

<ActorName> actor-name </ActorName> 
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<Gender> male </Gender> 
<Age> age </Age> 
<Www> web-page-url </Www> 
<Occurrence id=""> 
5 <Duration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2 ] ... </Location> 

<Motion> V,, Vy Vp </Motion> 

<Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
10 </Occurrence> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 

<Location> frame: [xl yl x2 y2] ... </Location> 

<Motion> Vx Vy Vj v^, Vp Vy </Motion> 
15 <Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 

</Character> 
2 0 <Character name=""> 

<ActorName> actor-name </ActorName> 

<Gender> male </Gender> 

<Age> age </Age> 

<Www> web-page-url </Www> 

2 5 ■ <Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
<Location> frame: [xl yl x2 y2] ... </Location> 
<Motion> Vx Vy Vj Vp v^ </Motion> 
<Text> text-annotation </Text> 

3 0 <Audio> voice-annotation </Audio> 

< /Occur rence> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
<Location> frame: [xl yl x2 y2 ] . . . * </Location> 

3 5 <Motion> Vx Vy v^ v^, Vp v^ </Motion> 

<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 
< /Occur rence> 

4 0 </Character> 

</CharacterProf ile> 



45 



The descriptor <CharacterProf ile> specifies the detailed 
information for certain characters in a program. Each 
character is specified by the descriptor <Character> with 
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a name attribute. Each occurrence of a character is 
specified by the descriptor <Occurrence> with an id 
attribute which may be matched with a clip id under 
<CloseUpView> . 



5 • Object profile 



<ObjectProf ile> 

<ObjectList> object-name . . . </Ob jectList> 
<Object name=""> 
10 <Www> web-page-url </Www> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
xQ <Location> frame: [xl yl x2 y2] ... </Location> 

Ln <Motion> Vy v„ Vp </Motion> 

^ 15 <Text> text-annotation </Text> 

"Tz <Audio> voice-annotation </Audio> 

x=s^ </Occurrence> 
fr\ <Occurrence id=""> 

2 <Duration> start-frame-id end-frame-id </Duration> 

Q 2 0 <Location> frame: [xl yl x2 y2] ... </Location> 

y= <Motion> Vy v^ Vc, Vp v^ </Motion> 

ly <Text> text-annotation </Text> 

f=Fs 

^ <Audio> voice-annotation </Audio> 

</Occurrence> 

~ 25 

</Object> 
<Object name=""> 

<Www> web-page-url </Www> 
<Occurrence id=""> 
3 0 <Duration> start-frame-id end-frame-id </Duration> 

<Location> frame; [xl yl x2 y2] ... </Location> 
<Motion> Vx Vy Vj v^, Vp Vy </Motion> 
<Text> text-annotation </Text> 
<Audio> voice-annotation </Audio> 

3 5 </Occurrence> 

<Occurrence id=""> 

<Duration> start-frame-id end-frame-id </Duration> 
<Location> frame: [xl yl x2 y2] ... </Location> 
<iyiotion> v^^ Vy v^ v^, Vp v^ </Motion> 

4 0 <Text> text-annotation </Text> 

<Audio> voice-annotation </Audio> 
</Occurrence> 
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</Object> 
</ObjectProf ile> 

5 The descriptor <0b j ectProf ile> specifies the detailed 

information for certain objects in a program. Each object 
is specified by the descriptor <Object> with a name 
attribute. Each occurrence of a object is specified by 
the descriptor <Occurrence> with an id attribute which 
10 may be matched with a clip id under <CloseUpView> . 

• Color profile 

<ColorProf ile> 
15 </ColorProf ile> 

The descriptor <ColorProf ile> specifies the detailed 
color information of a program. All MPEG-7 color 
descriptors may be placed under here. 

2 0 • Texture profile 

<TextureProf ile> 
< /Texture Prof ile> 

25 

The descriptor <TextureProf ile> specifies the detailed 
texture information of a program. All MPEG-7 texture 
descriptors may be placed under here. 

• Shape profile 

30 

<ShapeProf ile> 



</ShapeProf ile> 



40 

The descriptor <ShapeProf ile> specifies the detailed 
shape information of a program. All MPEG- 7 shape 
descriptors may be placed under here. 

• Motion profile 

5 

<MotionProf ile> 
</MotionProf ile> 

The descriptor <MotionProf ile> specifies the detailed 
motion information of a program. All MPEG-7 motion 
descriptors may be placed under here. 

User Description Scheme 

The proposed user description scheme includes three major 
sections for describing a user. The first section 
identifies the described user. The second section records 
a number of settings which may be preferred by the user. 
The third section records some statistics which may reflect 
certain usage patterns of the user. Therefore, the overall 
structure of the proposed description scheme is as follows: 

<?XML version="l . 0"> 

<!DOCTYPE MPEG-7 SYSTEM "mpeg-7 . dtd"> 
<UserIdentity> 

<UserID> . . . </UserID> 
2 5 <UserName> . . . </UserName> 
</UserIdentity> 
<UserPref erences> 

<BrowsingPref erences> . . . </BrowsingPref erences> 

<FilteringPreferences> . . . </FilteringPref erences> 
3 0 <SearchPreferences> ... </SearchPref erences> 

<DevicePref erences> . . . </DevicePref erences> 
</UserPref erences> 
<UserHistory> 

<BrowsingHistory> . . . </BrowsingHistory> 
3 5 <FilteringHistory> ... </FilteringHistory> 

<SearchHistory> . . . </SearchHistory> 
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<DeviceHistory> . . . </DeviceHistory> 
</UserHistory> 
<UserDemographics> 

<Age> . . . </Age> 
5 <Gender> . . . </Gender> 

<ZIP> . . . </ZIP> 
</UserDemographics> 



User Identity 



10 • User ID 



<UserID> user-id </UserID> 



o 



The descriptor <UserID> contains a number or a string to 

H 15 identify a user. 



m 



y J 
n 



• User name 

<UserNaine> user-name </UserName> 

20 The descriptor <UserName> specifies the name of a user. 

User Preferences 

• Browsing preferences 



<BrowsingPref erences> 

2 5 <Views> 

<ViewCategory id=""> view-id . . . </ViewCategory> 
<ViewCategory id=""> view-id . . . </ViewCategory> 

</Views> 

3 0 <FrameFrequency> frequency . . . <FrameFrequency> 

<ShotFrequency> frequency . . . <ShotFrequency> 
<KeyFrameLevel> level-id ... <Key Frame Level > 
<HighlightLength> length , . . <HighlightLength> 



3 5 </BrowsingPref erences> 



5 
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The descriptor <BrowsingPref erences> specifies the 
browsing preferences of a user. The user's preferred 
views are specified by the descriptor <Views>. For each 
category, the preferred views are specified by the 
5 descriptor <ViewCategory> with an id attribute which 

corresponds to the category id. The descriptor 
<FrameFrequency> specifies at what interval the frames 
should be displayed on a browsing slider under the frame 
view. The descriptor <ShotFrequency> specifies at what 
10 interval the shots should be displayed on a browsing 

slider under the shot view. The descriptor 
<KeyFrameLevel> specifies at what level the key frames 
should be displayed on a browsing slider under the key 
frame view. The descriptor <HighlightLength> specifies 
m 15 which version of the highlight should be shown under the 

^ highlight view. 

I 8 : 
III 

B • Filtering preferences 

m ■ 

ly <FilteringPreferences> 

2 0 <Categories> category-name ... </Categories> 

^ <Channels> channel -number . . . </Channels> 

<Ratings> rating-id . . . </Ratings> 

<Shows> show-name . . . </Shows> 

<Authors> author-name . . . </Authors> 

2 5 <Producers> producer-name ... </Producers> 

<Directors> director-name ... </Directors> 
<Actors> actor-name . . . </Actors> 
<Keywords> keyword . . . </Keywords> 
<Titles> title-text . . . </Titles> 

30 

</FilteringPref erences> 

The descriptor <FilteringPref erences> specifies the 
filtering related preferences of a user. 

3 5 • Search preferences 



OearchPref erences> 
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<Categories> category-name . . . </Categories> 
<Channels> channel -number . . . </Channels> 
<Ratings> rating-id . . . </Ratings> 
<Shows> show-name . . . </Shows> 
5 <Authors> author-name . . . </Authors> 

<Producers> producer-name . . . </Producers> 
<Directors> director-name . . . </Directors> 
<Actors> actor-name . , , </Actors> 
<Keywords> keyword . . . </Keywords> 
10 <Titles> title-text ... </Titles> 



</SearchPref erences> 



The descriptor <SearchPref erences> specifies the search 
15 related preferences of a user: 

• Device preferences 



Q <DevicePref erences> 

B <Brightness> brightness-value </Brightness> 

5 2 0 <Contrast> contrast-value </Contrast> 

<Volume> volume-value </Volume> 
</DevicePref erences> 

The descriptor <DevicePref erences> specifies the device 
25 preferences of a user. 



Usage History 
• Browsing history 



<BrowsingHistory> 
3 0 <Views> 

<ViewCategory id=""> view-id . . . </ViewCategory> 
<ViewCategory id=""> view-id . . . </ViewCategory> 

</Views> 

3 5 <FrameFrequency> frequency . . . <FrameFrequency> 

<ShotFrequency> frequency . . . <ShotFrequency> 
<KeyFrameLevel> level-id . . . <KeyFrameLevel> 
<HighlightLength> length . . . <HighlightLength> 
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</BrowsingHistory> 



The descriptor <BrowsingHistory> captures the history of 
a user»s browsing related activities. 



• Filtering history 



<FilteringHistory> 

<Categories> category-name . . . </Categories> 
10 <Channels> channel-number . . . </Channels> 

<Ratings> rating-id . . . </Ratings> 
<Shows> show-name . . . </Shows> 
O <Authors> author-name . . . </Authors> 

*^ <Producers> producer-name . . . </Producers> 

in 

15 <Directors> director-name . . . </Directors> 

pi <Actors> actor-name . . . </Actors> 

fQ <Keywords> keyword . . . </Keywords> 

□ <Titles> title-text . . . </Titles> 

«... 
3 2 0 </FilteringHistory> 

Q 

The descriptor <FilteringHistory> captures the history of 

gi a user's filtering related activities. 

^ * 

• Search history 

25 

<SearchHistory> 

<Categories> category-name . . . </Categories> 

<Channels> channel -number . . . </Channels> 

<Ratings> rating-id . . . </Ratings> 
3 0 <Shows> show-name ... </Shows> 

<Authors> author-name . . . </Authors> 

<Producers> producer-name . . . </Producers> 

<Directors> director-name . . . </Directors> 

<Actors> actor-name . . . </Actors> 
3 5 <Keywords> keyword . . . </Keywords> 

<Titles> title-text . . . </Titles> 

</SearchHistory> 



The descriptor <SearchHistory> captures the history of a 
user's search related activities. 



• Device history 

5 <DeviceHistory> 

<Brightness> brightness-value . . . </Brightness> 
<Contrast> contrast-value , . . </Contrast> 
<Volume> volume-value . . . </Vol\ame> 
</DeviceHistory> 

0 

The descriptor <DeviceHistory> captures the history of a 
user's device related activities. 

User demographics 

• Age 

5 

<Age> age </Age> 

The descriptor <Age> specifies the age of a user. 

• Gender 

0 

<Gender> . . . </Gender> 

The descriptor <Gender> specifies the gender of a user. 

• ZIP code 

5 

<zip> . . . </ZIP> 

The descriptor <ZIP> specifies the ZIP code of where a 
user lives. 



System Description Scheme 



The proposed system description scheme includes four major 
sections for describing a user. The first section 
identifies the described system. The second section keeps 
5 a list of all known users. The third section keeps lists of 
available programs. The fourth section describes the 
capabilities of the system. Therefore, the overall 
structure of the proposed description scheme is as follows: 

<?XML version="l. 0"> 

ilCbOCTYPE MPEG-7 SYSTEM "mpeg-7 . dtd"> 
<SystemIdentity> 

<SystemID> . . . </SystertiID> 

<SysteinNaine> . . . </SysteinName> 

<SystemSerialNiainber> . . . </SystemSerialNuinber> 
i/feystemldentity> 
<SystemUsers> 

<Users> . . . </Users> 
</SystemUsers> 
<SystemPrograins> 

2 0 <Categories> ... </Categories> 

<Channels> . . . </Channels> 

<Programs> . . . </Progranis> 
</SystemPrograins> 
<SysteinCapabilities> 
2 5 <Views> . . . </Views> 
</SystemCapabilities> 

System Identity 
• System ID 

30 

<SystemID> system-id </SystemID> 

The descriptor <SystemID> contains a number or a string 
to identify a video system or device. 



3 5 • System name 



<SystemName> sys tern- name </SystemNaine> 



The descriptor <SysteTTiName> specifies the name of a video 
system or device . 

5 • System serial nxxEtiber 

<SysteinSerialNuinber> system-serial-number </SystemSerialNumber> 

The descriptor <SystemSerialNumber> specifies the serial 
0 number of a video system or device. 

System Users 
• Users 

<Users> 
5 <User> 

<UserID> user-id </UserID> 
<UserName> user-name </UserName> 
</User> 
<User> 

0 <UserID> user-id </UserID> 

<UserName> user-name </UserName> 
</User> 

</Users> 

5 

The descriptor <SystemUsers> lists a number of users who 
have registered on a video system or device. Each user is 
specified by the descriptor <User> . The descriptor 
<UserID> specifies a number or a string which should 
0 match with the number or string specified in <UserID> in 

one of the user description schemes. 
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Programs in the System 



• Categories 

<Categories> 
5 <Category> 

<CategoryID> category-id </CategoryID> 
<CategoryName> category-name </CategoryName> 
<SubCategories> sub-category-id . . . </SubCategories> 
</Category> 
10 <Category> 

<CategoryID> category-id </CategoryID> 
<CategoryNan\e> category-name </CategoryName> 
<SubCategories> sub-category-id . . . </SubCategories> 
</Category> 

15 

</Categories> 

The descriptor <Categories> lists a number of categories 
which have been registered on a video system or device. 
20 Each category is specified by the descriptor <Category>. 

The major-sub relationship between categories is captured 
by the descriptor < SubCategories> . 

• Channels 

2 5 <Channels> 

<Channel> 

<ChannelID> channel-id </ChannelID> 
<ChannelName> channel-name </ChannelName> 
<SubChannels> sub-channel-id . . . </SubChannels> 

3 0 </Channel> 

<Channel> 

<ChannelID> channel-id </ChannelID> 
<ChannelName> channel-name </ChannelName> 
<SubChannels> sub-channel-id . . . </SubChannels> 
3 5 </Channel> 

</Channels> 



The descriptor <Channels> lists a number of channels 
40 which have been registered on a video system or device. 



Each channel is specified by the descriptor <Channel>. 
The major-sub relationship between channels is captured 
by the descriptor < SubChannels> . 



• Programs 

<Programs> 

<CategoryPrograms> 

<CategoryID> category-id </CategoryID> 

<Prograins> program-id . . . </Prograins> 
</CategoryPrograms> 
<CategoryPrograms> 

<CategoryID> category-id </CategoryID> 

<Programs> program-id . . . </Programs> 
</CategoryPrograms> 

<ChannelPrograms> 

<ChannelID> channel-id </ChannelID> 

<Programs> program-id . . . </Programs> . 
< /Channel Programs> 
<Channel Programs > 

<ChannelID> channel-id </ChannelID> 

<Programs> program-id . . . </Programs> 
< /Channel Programs> 

</Programs> 

The descriptor <Programs> lists programs who are 
available on a video system or device. The programs are 
grouped under corresponding categories or channels. Each 
group of programs are specified by the descriptor 
<CategoryPrograms> or < Channel Programs > . Each program id 
contained in the descriptor <Programs> should match with 
the number or string specified in <ProgramID> in one of 
the program description schemes. 



System Capabilities 



• Views 

<Views> 
5 <View> 

<ViewID> view-id </ViewID> 
<ViewName> view-name </ViewName> 
</View> 
<View> 

0 <ViewID> view-id </ViewID> 

<ViewName> view-name </ViewName> 
</View> 

</Views> 

5 

The descriptor <Views> lists views which are supported 
by a video system or device. Each view is specified by 
the descriptor <View>. The descriptor <ViewName> 
contains a string which should match with one of the 
0 following views used in the program description 

schemes: Thumbnail View, SlideView, FrameView, ShotView, 
KeyFrameView, HighlightView, Event View, and 
CloseUpView. 

5 The present inventors came to the realization 

that the program description scheme may be further 
modified to provide additional capabilities. Referring 
to FIG. 13, the modified program description scheme 400 
includes four separate types of information, namely, a 

0 syntactic structure description scheme 402, a semantic 
structure description scheme 404, a visualization 
description scheme 406, and a meta information 
description scheme 408. It is to be understood that in 
any particular system one or more of the description 

5 schemes may be included, as desired. 

Referring to FIG. 14, the visualization 
description scheme 4 06 enables fast and effective 
browsing of video program (and audio programs) by 



allowing access to the necessary data, preferably in a 
one- step process. The visualization description scheme 
406 provides for several different presentations of the 
video content (or audio) , such as for example, a 
thumbnail view description scheme 410, a key frame view 
description scheme 412, a highlight view description 
scheme 414, an event view description scheme 416, a 
close-up view description scheme 418, and an alternative 
view description scheme 420. Other presentation 
techniques and description schemes may be added, as 
desired. The thumbnail view description scheme 410 
preferably includes an image 422 or reference to an image 
representative of the video content and a time reference 
424 to the video. The key frame view description scheme 
412 preferably includes a level indicator 42 6 and a time 
reference 428. The level indicator 426 accommodates the 
presentation of a different number of key frames for the 
same video portion depending on the user's preference. 
The highlight view description scheme 414 includes a 
length indicator 430 and a time reference 432. The 
length indicator 430 accommodates the presentation of a 
different highlight duration of a video depending on the 
user's preference. The event view description scheme 416 
preferably includes an event indicator 434 for the 
selection of the desired event and a time reference 436. 
The close-up view description scheme 418 preferably 
includes a target indicator 438 and a time reference 440. 
The alternate view description scheme preferably includes 
a source indicator 442 . To increase performance of the 
system it is preferred to specify the data which is 
needed to render such views i^n a centralized and 
straightforward manner. By doing so, it is then feasible 
to access the data in a simple one-step process without 
complex parsing of the video. 

Referring to FIG. 15, the meta information 
description scheme 4 08 generally includes various 
descriptors which carry general information about a video 
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(or audio) program such as the title, category, keywords, 
etc. Additional descriptors, such as those previously 
described, may be included, as desired. 

Referring again to FIG. 13, the syntactic 
structure description scheme 402 specifies the physical 
structure of a video program (or audio), e.g., a table of 
contents. The physical features, may include for 
example, color, texture, motion, etc. The syntactic 
structure description scheme 402 preferably includes 
three modules, namely a segment description scheme 4 50, a 
region description scheme 452, and a segment /region 
relation graph description scheme 454. The segment 
description scheme 450 may be used to define 
relationships between different portions of the video 
consisting of multiple frames of the video. A segment 
description scheme 450 may contain another segment 
description scheme 450 and/or shot description scheme to 
form a segment tree. Such a segment tree may be used to 
define a temporal structure of a video program. Multiple 
segment trees may be created and thereby create multiple 
table of contents. For example, a video program may be 
segmented into story units, scenes, and shots, from which 
the segment description scheme 450 may contain such 
information as a table of contents. The shot description 
scheme may contain a number of key frame description 
schemes, a mosaic description scheme (s), a camera motion 
description scheme (s), etc. The key frame description 
scheme may contain a still image description scheme which 
may in turn contains color and texture descriptors. It 
is noted that various low level descriptors may be 
included in the still image description scheme- under the 
segment description scheme. Also, the visual descriptors 
may be included in the region description scheme which is 
not necessarily under a still image description scheme. 
On example of a segment description scheme 450 is shown 
in FIG. 16. 



53 

Referring to FIG. 17, the region description 
scheme 452 defines the interrelationships between groups 
of pixels of the same and/or different frames of the 
video. The region description scheme 452 may also 
5 contain geometrical features, color, texture features, 
motion features, etc. 

Referring to FIG. 18, the segment /region 
relation graph description scheme 454 defines the 
interrelationships between a plurality of regions (or 
10 region description schemes) , a plurality of segments (or 
segment description schemes) and/or a plurality of 
regions (or description schemes) and segments (or 
□ description schemes) . 

Referring again to FIG. 13, the semantic 
^ 15 structure description scheme 404 is used to specify 

2 semantic features of a video program (or audio), e.g. 

Q semantic events. In a similar manner to the syntactic 

53 structure description scheme, the semantic structure 

description scheme 404 preferably includes three modules, 
yi 2 0 namely an event description scheme 48 0, an object 

Ir description scheme 482, and an event/objection relation 

p graph description scheme 484. The event description 

G scheme 480 may be used to form relationships between 

different events of the video normally consisting of 
25 multiple frames of the video. An event description 

scheme 480 may contain another event description scheme 
480 to form a segment tree. Such an event segment tree 
may be used to define a semantic index table for a video 
program. Multiple event trees may be created and thereby 
3 0 creating multiple index tables. For example, a video 

program may include multiple events, such as a basketball 
dunk, a fast break, and a free throw, and the event 
description scheme may contain such information as an 
index table. The event description scheme may also 
3 5 contain references which link the event to the 

corresponding segments and/or regions specified in the 
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syntactic structure description scheme. On example of an 
event description scheme is shown in FIG. 19. 

Referring to FIG. 20, the object description 
scheme 4 82 defines the interrelationships between groups 
of pixels of the same and/or different frames of the 
video representative of objects. The object description 
scheme 4 82 may contain another object description scheme 
and thereby form an object tree. Such an object tree may 
be used to define an object index table for a video 
program. The object description scheme may also contain 
references which link the object to the corresponding 
segments and/or regions specified in the syntactic 
structure description scheme. 

Referring to FIG. 21, the event /object relation 
graph description scheme 484 defines the 
interrelationships between a plurality of events (or 
event description schemes) , a plurality of objects (or 
object description schemes) , and/or a plurality of events 
(or description schemes) and objects (or description 
schemes) . 

After further consideration, the present 
inventors came the realization that the particular design 
of the user preference description scheme is important to 
implement portability, while permitting adaptive 
updating, of the user preference description scheme. 
Moreover, the user preference description scheme should 
be readily usable by the system while likewise being 
suitable for modification based on the user » s historical 
usage patterns. It is possible to collectively track all 
users of a particular device to build a database for the 
historical viewing preferences of the users of the 
device, and thereafter process the data dynamically to 
determine which content the users would likely desire. 
However, this implementation would require the storage of 
a large amount of data and the associated dynamic 
processing requirements to determine the user 
preferences. It is to be understood that the user 



55 

preference description scheme may be used alone or in 
combination with other description scheme. 

Referring to FIG. 22, to achieve portability 
and potentially decreased processing requirements the 
5 user preference description scheme 2 0 should be divided 

into at least two separate description schemes, namely, a 
usage preference description scheme 500 and a usage 
history description scheme 502. The usage preference 
description scheme 500, described in detail later, 
10 includes a description scheme of the user's audio and/or 
video consumption preferences. The usage preference 
description scheme 500 describes one or more of the 
□ following, depending on the particular implementation, 

fd (a) browsing preferences, (b) filtering preferences, (c) 

B 15 searching preferences, and (d) device preferences of the 

user. The type of preferences shown in the usage 
preference description scheme 500 are generally 
immediately usable by the system for selecting and 
otherwise using the available audio and/or video content. 
In 2 0 In other words, the usage preference description scheme 

nj 

500 includes data describing audio and/or video 

y i 

O consumption of the user. The usage history description 

~ scheme 502, described in detail later, includes a 

description scheme of the user's historical audio and/or 

25 video activity, such as browsing, device settings, 

viewing, and selection. The usage history description 
scheme 502 describes one or more of the following, 
depending on the particular implementation, (a) browsing 
history, (b) filtering history, (c) searching history, and 

3 0 (d) device usage history. The type of preferences shown 
in the usage history description scheme 502 are not 
generally immediately usable by the system for selecting 
and otherwise using the available audio and/or video 
content. The data contained in the usage history 

3 5 description scheme 502 may be considered generally 
"unprocessed", at least in comparison to the data 
contained in the usage preferences description scheme 500 



m 



rp 
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because it generally contains the historical usage data 
of the audio and/or video content of the viewer. 

In general, capturing the user^s usage history- 
facilitates "automatic" composition of user preferences by 
a machine, as desired. When updating the user preference 
description scheme 500 it is desirable that the usage 
history description scheme 502 be relatively symmetric to 
the usage preference description scheme 500. The 
symmetry permits more effective updating because less 
interpretation between the two description schemes is 
necessary in order to determine what data should be 
included in the preferences. Numerous algorithms can 
then be applied in utilization of the history information 
in deriving user preferences. For instance, statistics 
can be computed from the history and utilized for this 
purpose . 

After consideration of the usage preference 
description 500 and the usage history description 502, 
the present inventors came to the realization that in the 
home environment many different users with different 
viewing and usage preferences may use the same device. 
For example, with a male adult preferring sports, a 
female adult preferring afternoon talk shows, and a three 
year old child preferring children's programming, the 
total information contained in the usage preference 
description 500 and the usage history description 502 
will not be individually suitable for any particular 
user. The resulting composite data and its usage by the 
device is frustrating to the users because the device 
will not properly select and present audio and/or video 
content that is tailored to any particular user. To 
alleviate this limitation, the user preference 
description 20 may also include a user identification 
(user identifier) description 504. The user 
identification description 504 includes an identification 
of the particular user that is using the device. By 
incorporating a user identification description 504 more 



than one user may use the device while maintaining a 
different or a unique set of data within the usage 
preference description 500 and the usage history- 
description 502 . Accordingly, the user identification 
description 504 associates the appropriate usage 
preference description (s) 500 and usage history 
description (s) 502 for the particular user identified by 
the user identification description 504 . With multiple 
user identification descriptions 504, multiple entries 
within a single user identification description 504 
identifying different users, and/or including the user 
identification description within the usage preference 
description 500 and/or usage history description 502 to 
provide the association therebetween, multiple users can 
readily use the same device while maintaining their 
individuality. Also, without the user identification 
description in the preferences and/or history, the user 
may more readily customize content anonymously. In 
addition, the user's user identification description 504 
may be used to identify multiple different sets of usage 
preference descriptions 500 -- usage history (descriptions 
502, from which the user may select for present 
interaction with the device depending on usage 
conditions. The use of multiple user identification 
descriptions for the same user is useful when the user 
uses dultiple different types of devices, such as a 
television, a home stereo, a business television, a hotel 
television, and a vehicle audio player, and maintains 
multiple different sets of preference descriptions. 
Further, the identification may likewise be used to 
identify groups of individuals, such as for example, a 
family. In addition, devices that are used on a 
temporary basis, such as those in hotel rooms or rental 
cars, the user identification requirements may be 
overridden by employing a temporary session user 
identification assigned by such devices. In applications 
where privacy concerns may be resolved or are otherwise 
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not a concern, the user identification description 504 
may also contain demographic information of the user. In 
this manner, as the usage history description 502 
increases during use over time, this demographic data 
and/or data regarding usage patterns may be made 
available to other sources. The data may be used for any 
purpose, such as for example, providing targeted 
advertising or programming on the device based on such 
data . 

Referring to FIG. 23, periodically an agent 510 
processes the usage history description (s) 502 for a 
particular user to "automatically" determine the 
particular user's preferences. In this manner, the 
user's usage preference description 500 is updated to 
reflect data stored in. the usage history description 502. 
This processing by the agent 510 is preferably performed 
on a periodic basis so that during normal operation the 
usage history description 502 does not need to be 
processed, or otherwise queried, to determine the user's 
current browsing, filtering, searching, and device 
preferences. The usage preference description 500 is 
relatively compact and suitable for storage on a portable 
storage device, such as a smart ,card, for use by other 
devices as previously described. 

Frequently, the user may be traveling away from 
home with his smart card containing his usage preference 
description 500. During such traveling the user will 
likely be browsing, filtering, searching, and setting 
device preferences of audio and/or video content on 
devices into which he provided his usage preference 
description 500. However, in some circumstances the 
audio and/or video content browsed, filtered, searched, 
and device preferences of the user may not be typically 
what he is normally interested in. In addition, for a 
single device the user may desire more than one profile 
depending on the season, such as football season, 
basketball season, baseball season, fall, winter, summer. 
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and spring. Accordingly, it may not be appropriate for 
the device to create a usage history description 502 and 
thereafter have the agent 510 "automatically" update the 
user*s usage preference description 500. This will in 
5 effect corrupt the user's usage preference description 
500. Accordingly, the device should include an option 
that disables the agent 510 from updating the usage 
preference description 500. Alternatively, the usage 
preference description 500 may include one or more fields 
10 or data structures that indicate whether or not the user 
desires the usage preference description 500 (or portions 
thereof) to be updated. 
O Refierring to FIG. 24, the device may use the 

program descriptions provided by any suitable source 
K 15 describing the current and/or future audio and/or video 

^ content available from which a filtering agent 52 0 

O selects the appropriate content for the particular 

^ user(s). The content is selected based upon the usage 

Q preference description for a particular user 

yl 20 ident if ication (s) to determine a list of preferred audio 

fS and/or video programs. 

O As it may be observed, with a relatively 

^ compact user preference description 500 the user's 

preferences are readily movable to different devices, 

2 5 such as a personal video recorder, a TiVO player, a 

RePlay Networks player, a car audio player, or other 
audio and/or video appliance. Yet, the user preference 
description 50 0 may be updated in accordance with the 
user's browsing, filtering, searching, and device 

3 0 preferences . 

Referring to FIG. 25, the usage preference 
description 500 preferably includes three different 
categories of descriptions, depending on the particular 
implementation. The preferred descriptions include (a) 
35 browsing preferences description 530, (b) filtering and 
search preferences description, 532 and (c) device 
preferences description 534. The browsing preferences 



description 53 0 relates to the viewing preferences of 
audio and/or video programs. The filtering and search 
preferences description 532 relates to audio and/or video 
program level preferences. The program level preferences 
are not necessarily used at the same time as the 
(browsing) viewing preferences. For example, preferred 
programs can be determined as a result of filtering 
program descriptions according to user's filtering 
preferences. A particular preferred program may 
subsequently be viewed in accordance with user's browsing 
preferences. Accordingly, efficient implementation may 
be achieved if the browsing preferences description 530 
is separate, at least logically, from the filtering and 
search preferences description 532 . The device 
preferences description 534 relates to the preferences 
for setting up the device in relation to the type of 
content being presented, e.g. romance, drama, action, 
violence, evening, morning, day, weekend, weekday, and/or 
the available presentation devices. For example, 
presentation devices may include stereo sound, mono 
sound, surround sound, multiple potential displays, 
multiple different sets of audio speakers, AC-3, and 
Dolby Digital. It may likewise be observed that the 
device preferences description 534 is likewise separate, 
at least logically, from the browsing description 530 and 
f iltering/search preferences description 532. 

The browsing preferences description 53 0 
contains descriptors that describe preferences of the 
user for browsing multimedia (audio and/or video) 
information. In the case of video, for example, the 
browsing preferences may include user's preference for 
continuous playback of the entire program versus 
visualizing a short summary of the program. Various 
summary types may be described in the program 
descriptions describing multiple different views of 
programs where these descriptions are utilized by the 
device to facilitate rapid non-linear browsing, viewing. 
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and navigation. Parameters of the various summary types 
should also be specified, i.e., number of hierarchy 
levels when the keyframe summary is preferred, or the 
time duration of the video highlight when highlight 
summary is preferred. In addition, browsing preferences 
may also include descriptors describing parental control 
settings. A switch descriptor (set by the user) should 
also be included to specify whether or not the 
preferences can be modified without consulting the user 
first. This prevents inadvertent changing or updating of 
the preferences by the device. In addition, it is 
desirable that the browsing preferences are media content 
dependent. For example, a user may prefer 15 minute 
video highlight of a basketball game or may prefer to see 
only the 3 -point shots. The same user may prefer a 
keyframe summary with two levels of hierarchy for home 
videos . 

The filtering and search preferences 
description 532 preferably has four descriptions defined 
therein, depending on the particular embodiment. The 
keyword preferences description 540 is used to specify 
favorite topics that may not be captured in the title, 
category, etc., information. This permits the acceptance 
of a query for matching entries in any of the available 
data fields. The content preferences description 542 is 
used to facilitate capturing, for instance, favorite 
actors, directors. .The creation preferences description 
544 is used to specify capturing, for instance, titles of 
favorite shows. The classification preferences 
description 546 is used to specify descriptions, for 
instance, a favorite program category. A switch 
descriptor, activated by the user, may be included to 
specify whether or not the preferences may be modified 
without consulting the user, as previously described. 

The device preferences description 534 contains 
descriptors describing preferred audio and/or video 
rendering settings, such as volume, balance, bass. 
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treble, brightness, contrast, closed captioning, AC- 3, 
Dolby digital, which display device of several, type of 
display device, etc. The settings of the device relate 
to how the user browses and consumes the audio and/or 
video content. It is desirable to be able to specify the 
device setting preferences in a media type and content- 
dependent manner. For example the preferred volume 
settings for an action movie may be higher than a drama, 
or the preferred settings of bass for classical music and 
rock music may be different. A switch descriptor, 
activated by the user, may be included to specify whether 
or not the preferences may be modified without consulting 
the user, as previously described. 

Referring to FIG. 26, the usage preferences 
description may be used in cooperation with an MPEG-7 
compliant data stream and/or device. MPEG-7 descriptions 
are described in ISO/IEC JTC1/SC29/WG11 "MPEG-7 Media/Meta 
DSs (V0.2), August 1999, incorporated by reference 
herein. It is preferable that media content descriptions 
are consistent with descriptions of preferences of users 
consuming the media. Consistency can be achieved by 
using common descriptors in media and user preference 
descriptions or by specifying a correspondence between 
user preferences and media descriptors. Browsing 
preferences descriptions are preferably consistent with 
media descriptions describing different views and 
summaries of the media. The content preferences 
description 542 is preferably consistent with, e.g., a 
subset of the content description of the media 553 
specified in MPEG-7 by content description scheme. The 
classification preferences description 544 is preferably 
consistent with, e.g., a subset of the classification 
description 554 defined in MPEG-7 as classification 
description scheme. The creation preferences description 
546 is preferably consistent with, e.g., a subset of the 
creation description 556 specified in MPEG-7 by creation 
description scheme. The keyword preferences description 



540 is preferably a string supporting multiple languages 
and consistent with corresponding media content 
description schemes. Consistency between media and user 
preference descriptions is depicted or shown in FIG. 2 6 
by couble arrows in the case of content, creation, and 
classification preferences. 

Referring to FIG. 27, the usage history 
description 502 preferably includes three different 
categories of descriptions, depending on the particular 
implementation. The preferred descriptions include (a) 
browsing history description 560, (b) filtering and 
search history description 562, and (c) device usage 
history description 564, as previously described in 
relation to the usage preference description 500. The 
filtering and search history description 562 preferably 
has four descriptions defined therein, depending on the 
particular embodiment, namely, a keyword usage history 
description 566, a content usage history description 568, 
a creation preferences description 570, and a 
classification usage history description 572, as 
previously described with respect to the preferences. 
The usage history description 502 may contain additional 
descriptors therein (or description if desired) that 
describe the time and/or time duration of information 
contained therein. The time refers to the duration of 
consuming a particular audio and/or video program. The 
duration of time that a particular program has been 
viewed provides information that may be used to determine 
user preferences. For example, if a user only watches a 
show for 5 minutes then it may not be a suitable 
preference for inclusion the usage preference description 
500. In addition, the present inventors came to the 
realization that an even more accurate measure of the 
user*s preference of a particular audio and/or video 
program is the time viewed in light of the total duration 
of the program. This accounts for the relative viewing 
duration of a program. For example watching 3 0 minutes 
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of a 4 hour show may be of less relevance than watching 
3 0 minutes of a 30 minute show to determine preference 
data for inclusion in the usage preference description 
500 . 

5 Referring to FIG. 28, an exemplary example of 

an audio and/or video program receiver with persistent 
storage is illustrated. As shown, audio/video program 
descriptions are available from the broadcast or other 
source, such as a telephone line. The user preference 
10 description facilitate personalization of the browsing, 
filtering and search, and device settings. In this 
embodiment, the user preferences are stored at the user's 
y terminal with provision for transporting it to other 

yi systems, for example via a smart card. Alternatively, 

K 15 the user preferences may be stored in a server and the 

5 content adaptation can be performed according to user 

descriptions at the server and then the preferred content 
is transmitted to the user. The user may directly 
provide the user preferences, if desired. The user 

2 0 preferences and/or user history may likewise be provided 
to a service provider. The system may employ an 

B application that records user's usage history in the form 

O 

of usage history description, as previously defined. The 
usage history description is then utilized by another 
25 application, e.g., a smart agent, to automatically map 
usage history to user preferences. 

Additional Attributes and Descriptors 
In The Description and The Description Scheme 
30 The present inventors came to the realization 

that additional functionality for the system may be 
achieved by the incorporation of particular types of 
information in the descriptions and description schemes. 
A description scheme is a data model of descriptions. It 

3 5 specifies the descriptors and their syntax as they are 

used in the description. In what follows, use the terms 
description and description scheme may be used 
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interchangeably since they both correspond to describing 
media and user preferences. An explanation of the 
additional attributes and descriptors in the descriptions 
will be provided, followed by an example of portions of 
5 example descriptions. 

After further consideration, there is a need 
for many users to maintain multiple separate user 
preference descriptions. Multiple user preference 
descriptions may correspond to, for example, different 
10 locations (e.g., at home, at the office, away from home, 
stationary versus traveling in a vehicle) , different 
situations, different times (e.g., different days, 
^ different seasons) , different emotional states of the 

m user (e.g., happy mood versus tired or sad), and/or 

2 15 persistence (e.g., temporary usage versus permanent 

^ usage) . Further, the user preference descriptions may 

B include differentiation for different terminals with 

P different primary functionalities (e.g., a personal video 

Q recorder versus a cell phone) . In addition, available 

^! 2 0 communication channel bandwidth at different locations or 

yl situations may use different preferences. Also, the 

2 preference of a user for the length of an audiovisual 

"~ summary of a video program for downloading may be 

different. The user in different usage conditions may 
25 use the user identification description scheme as a basis 
to distinguish between different devices and/or services. 
An example of different conditions may include a 
television broadcast receiver and a cellular telephone. 
In addition to maintaining multiple user 
30 preferences for a particular user based on the 

aforementioned conditions, the present inventors also 
came to the realization that the different locations, 
different situations, different emotional states, 
different seasons, and/or different terminals (etc.), may 
35 likewise be used as the basis for distinguishing between 
the user preference descriptions. 
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One technique to permit a particular user to 
have multiple preference descriptions and distinguishing 
them from one another is by using different usernames or 
by using a versioning mechanism, such as a version 
5 descriptor in the identification description scheme, as 
described later. 

As previously described, the system may include 
multiple user preference descriptions for a particular 
user. With multiple descriptions, the system may express 

10 the different user preferences with different 

granularity, e.g., a greater or lesser amount of detail. 
The increased granularity (sparseness) may be merely the 
result of applying a filter to the user preference 
description that further reduces the amount of data. In 

15 other words, the structure of the usage preference 

description may be identical with the difference being 
the result of the filter further reducing the data. In 
another embodiment, the variable granularity results in a 
different size of the data contained in the user 

20 preferences, which may be based upon, if desired, the 
location and/or application of the user. User 
preferences with increased granularity may be especially 
suitable for storage on portable memory devices with 
limited memory capability. Likewise, the granularity may 

25 be applied to the usage history. 

Another aspect of the present invention permits 
the user preferences (and history) to be based upon the 
media type, media source, or content (e.g., music versus 
video, radio versus television broadcast, and/or sports 

3 0 video versus home video) . These preferences relate to 
the audio and/or video itself, as opposed to a third 
party characterization of the desirability of the 
multimedia. The inclusion of this information permits a 
reduction in the computational processing requirements 

3 5 depending on the media type, media source, and/or content 
of the media. 
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Another feature that may be included in the 
system is a protection attribute for each, or a selected 
set of, component of the user descriptions. The 
protection attribute-s- specifies the access right of a 
5 system or service provider, typically a party other than 
the user himself, to the user's descriptions or any 
component thereof. In one embodiment, the protection 
attributes may be specified by a binary value that 
indicates the user's desire to permit others access to 

10 such data. One technique to implement the protection 
attribute is to include a protection attribute as a 
primitive attribute that is contained by all relevant 
parts of the user description scheme. 

Descriptors and description schemes for 

15 browsing preferences may be aligned with particular 

types of multimedia summary description schemes that are 
contained in ISO/IEC JTC1/SC2 9/WGll N3246, "MPEG-7 
Generic AV Description Schemes, Working Draft v2.0", 
Noordwi jkerhout , March 2000. This allows the user to 

20 specify the type of a particular visual summary of an 

audiovisual program, and the duration of a summary that 
is in the form of a visual highlight. However, after 
further consideration the present inventors have 
determined that specification of the preferred minimum 

2 5 and maximum amount of data permitted in an audiovisual 

summary significantly enhances the system capability. 
Such a provision provides, for example, the capability of 
the user effectively browsing audiovisual summaries of 
content over channels with limited bandwidth and using 
30 terminals with different limitations. With a terminal 
connected to a bandwidth limited channel, the user may 
specify preference for a relatively short highlight of 
the program, while with a terminal that is connected to a 
higher bandwidth channel, the user may specify preference 

3 5 for a longer highlight of the program. Such a set of 

channels may' be mobile channels and cable channels. In 
addition, for terminals that are not capable of 
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displaying frames at a video rate, the user may prefer 
keyframe summaries consisting of a maximum number of 
keyframes appropriate for the communication channel 
bandwidth. To achieve these enhancements, the present 
5 inventors propose using descriptors in the browsing 

preferences description (and description scheme, or other 
preferences description) specifying the minimum, maximum, 
and exact number of keyframes, and minimum, maximum, and 
exact duration of audio and/or visual highlights. 
10 As described, the description scheme is 

adaptable to express the preferred minimum and maximum 
amount of visual material to adapt to different viewing 
Q preferences as well as terminal and communication channel 

as £ 

bandwidth limitations. This implementation may be 
S 15 achieved by the following descriptors included in the 

S browsing preferences description scheme: 

□ MaxNumOf Keyframes , MinNumOf Keyframes , NumOf Keyframes , 

MaxSummaryDuration, MinSummaryDurat ion, and 
Q SummaryDuration . The MaxNumOf Keyframes and 

y1 2 0 MinNumof Keyframes preference descriptors specify, 

^ respectively, the maximum and minimum number of keyframes 

O in the keyframe -summary of a video program. Depending on 

O 

the known bandwidth conditions of a known connection that 
the user uses regularly, he or she may specify these 

25 descriptors. The MaxSummaryDuration and 

MinSummaryDurat ion descriptors specify, respectively, the 
maximum and minimum temporal duration of an audiovisual 
highlight summary. Again, depending on user's taste, 
terminal, and channel limitations, the user may specify 

3 0 these descriptors. The MaxSummaryDuration and 

MinSummaryDurat ion descriptors apply to preferences for 
audio signals as well as where audio highlights may have 
been generated by audio skimming methods. User's 
browsing preference descriptions may be correlated with 

35 media descriptions by a filtering agent 520 in Fig. 24 in 
order to determine media descriptions that contain 
summary descriptions that match user's preference 



descriptions and provide the user the associated 
summarized media in the preferred type of summary. 

An additional descriptor that may be introduced 
is an abstraction fidelity descriptor for universal 
multimedia access application, where fidelity of a 
summary abstraction of a program is described. This can 
correspond to the variation fidelity descriptor defined 
in ISO/IEC JTC1/SC29 WGll N3246, "MPEG-7 Multimedia 
Description Schemes, Working Draft v2.0", 
Noordwi jkerhout , March 2000. This provides an 
alternative to the explicit specification of the duration 
and bounds on the number of keyframes. A Segment Theme 
descriptor (s) may describe the preferred theme, or point 
of view, of a segment, e.g., a video or audio clip, 
annotated with its theme or emphasis point. For example, 
the theme may specify characteristics of the content of 
the theme. Such characterization may include a goal from 
your favorite team, 3 -point shots from your favorite 
player, etc. Specifying these descriptor (s) and also 
ranking them enables a client application or a server to 
provide to the user segments according to preferred 
themes (and/or their ranking) matching to the their 
labels or descriptors at the segment level, or provide 
users with pre-assembled highlights composed of segments 
with labels matching the SegmentTheme preference. 

Existing filtering and search user preference 
descriptions are directed to techniques of using the 
audiovisual content in an effective manner by finding, 
selecting and consuming the desired audiovisual material, 
while focusing on the content of the audiovisual 
materials. While such descriptions are beneficial, the 
present inventors came to the further realization that 
the identification of the source of the material, in 
contrast to merely its content, provides beneficial 
information for the processing and presentation of the 
audiovisual materials. For example, the source of the 
content may be from terrestrial sources, digital video 



disc, cable television, analog broadcast television, 
digital broadcast television, analog radio broadcasts, 
and digital radio broadcasts. The inclusion of this 
information permits the user to select among these 
different sources and increase effectiveness by narrowing 
down the choices to those sources that are available to 
the user, such as terrestrial broadcast which is more 
widely available than satellite broadcast. For example, 
user may describe user's preference for "Star Trek" 
episodes that are available from terrestrial broadcast 
channels only. 

This source distinction and identification may 
be performed by including a source preferences 
description scheme under the filtering and search 
preferences description scheme (or other description 
scheme) . Accordingly, the search and preferences 
description scheme may include from zero or one (or more 
if desired) source preferences description scheme. The 
source preferences description scheme may be derived from 
the Media Format description scheme or Publication 
Description Scheme specified in ISO/IEC JTC1/SC29/WG11 
N3247, MPEG-7 Multimedia Description Schemes, 
Experimentation Model (v2.0) Noordwi j kerhout , March 
2000 . 

Another feature that may be included in the 
system, in addition to the user's preferences, is the 
user's negative preferences. The negative preferences 
may include the user's dislikes and their relative rankings. By specifying 
the negative preferences, the system is less likely to select such matching preferences. 
This may be implemented, for example, by permitting positive and negative values to 
the preferencevalue descriptor. 

Another feature that may be included in the system is the specification 
of the user's preferences as a relative preference measure of a particular set of user 
preferences with respect to another set of preferences, such as for example, by using 
BetterThan and WorseThan descriptors. This permits an implicit relative ranking of 
preferences even in the absence of a preference value descriptor for each preference 



set. This may be implemented, for example, by including Betterthan and WorseThan 
descriptors in the filtering and search preferences descriptions. 

Expression of the Additional Attributes 
The following descriptions are expressed in XML (Extensible Markup 
Language), incorporated by reference herein. It is to be understood that any other 
description language may likewise be used. 

The definition of the user preference description may be as follows. 
<UserPreference> 

<UserIdentifier protection="true" userName="paurV> 
<UsagePreferences allowAutomaticUpdate="false"> 
<BrowsingPreferences> 

<VBrowsingPreferences> 
<FilteringAndSearchPreferences> 

</FilteringAndSearchPreferences> 
<DevicePreferences> 

</DevicePreferences> 
</UsageHistory> 

</UsageHistory> 
</UserPreference> 

The primitive attributes "protection" and "allow AutomaticUpdate" 
may be instantiated in the Userldentifier, Usage Preferences, and Usage History 
descriptions and all its relevant parts, namely, in Browsing Preferences description. 
Filtering and Search Preferences description, Device Preferences description, and sub- 
description schemes of the Usage History description Scheme. 

The "allow AutomaticUpdate" attribute (set by the user) should be 
included in a description scheme specifying whether or not the preferences can be 
automatically modified (e.g., by an agent utilizing the usage history description) 
without consulting with the user. 
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The protection attribute should be included in a description specifying 
whether the user allows the system to make preference/history public or not. When 
the user agrees to make some parts of his preference/history public, for example, to 
service providers, the service providers can collect this information and then serve to 
the user contents that are tailored to the user's history/preferences. In the above 
example description, the user prefers to keep his usemame private. He also does not 
wish the system to automatically update his preferences. 

The user identification description serves the purpose of an identifier 
that distinguishes a particular instantiation of the user description scheme fi*om other 
instantiations for other users or other instantiations for the same user for different 
usage conditions and situations. 

The usemame descriptor may identify a specific user firom other users. 
In a home setting, each member of the household may be identified using a usemame 
that is unique in the household for all devices that the members of that household use 
on a regular basis. A usemame can also be used to distinguish the user description 
scheme of not only an individual but also a group of people, e.g., the family. Those 
devices that are used on a temporary basis, potentially by many different people, (such 
as those in hotel rooms or rental cars) may assign temporary session identifications to 
ensure uniqueness of identifications. 

Alternatively, a version descriptor may also be included in the user 
identifier description to define different versions of the user descriptions (preferences 
and usage history) associated with a particular usemame. Through the mechanism of 
the version, a person can specify different preferences and usage history, 
corresponding to different locations (at home, at the office, away from home, 
stationary versus traveling in a vehicle), different situations, different emotional states 
(happy versus sad), different seasons, etc. Different user descriptions are 
distinguished by distinct version descriptors. The type of the version descriptor, may 
be for example, an integer, a string, or expressed as an attribute of the user 
identification description scheme. 

The usage preference description may include a PreferenceType 
description, distinguishing a particular set of preferences or history according to time. 



or place, or a place and time combination. The definition of the usage preference 
description may be as shown in the following example, where place is "office" and 
time period is "8 hours starting fi*om 8 AM" 

<PreferenceType> 
<Place> 

<PlaceName xml : lang="en">Offlce</PlaceName> 
<yPlace> 
<Time> 
<TimePoint> 
<h>8</h> 
</TimePoint> 
<Duration> 
<No_h>8</No_h> 
</Duration> 
</Time> 
</PrefereiiceType> 

The preferencetype descriptor may be used to identify the preference 
type of one or more set of preferences. As previously described, a user may have 
different preferences depending on the user's situation, location, time, season, and so 
on. 

The browsing preferences description may describe preferences of the 
user for browsing multimedia information. In essence, this description expresses the 
user's preferences for consuming (viewing, listening) a multimedia information. This 
browsing preferences description may include for example, a Summary Preferences 
description. The browsing preferences description may include in the case of video, 
for example, the user's preferences for continuous playback of the entire program 
versus visualizing a short summary of the program. Various summary types are 
specified in the Summary Description Scheme in ISO/IEC JTC1/SC29 WGl 1 N3246, 
"MPEG-7 Multimedia Description Schemes, Working Draft v2.0", Noordwijkerhout, 
March 2000, including a keyfi-ame simmiary, a highlight summary, etc., where 



parameters of the various summary types may also be specified by summary 
descriptions, e.g., the time duration of the video highlight summary. 

The browsing preferences description scheme may include one or more 
of the following non-exhaustive list of descriptors and descriptions in its description 
scheme. 

(A) The minimum number of keyframes (MinNumOQCeyframes) 
and the maximum number of keyframes (MaxNumOfKeyframes) 
descriptors may be included. These descriptors specify the user's 
preference for minimum and maximum number of frames in a 
keyframe summary of an audiovisual program, A user can specify 
these descriptors according to personal taste, situation, etc., and 
according to channel bandwidth and terminal resource limitation. 

(B) The minimum duration (MinSimmiaryDuration) and the 
maximum duration (MaxSummaryDuration) descriptors may be 
included. These descriptors specify the user's preference for the length 
of a highlight summary composed of key clips in the video. These 
descriptors may also, for example, be applied to an audio-only 
material. A user can specify these descriptors according to personal 
taste, situation, etc., and according to channel bandwidth and terminal 
resource limitations. 

An example for Summary Preferences description that can be 

included in usage preferences description is provided below. 

<yUsagePreferences> 
<;/BrowsmgPreferences> 
<SummaryPreferences> 

<SummaiyTypePreference>keyVideoClips<7SummaryTypePreference> 
<MinSummaryDuration><m>3<;/m><s>20</s></MinSummaiyDurati 
<MaxSuiTimaryDiiration><m>6<7m><s>40<Vs></MaxSui]m 
</SiimmaryPreferences> 
</BrowsmgPreferences> 
</UsagePreferences> 



(C) The abstraction fidelity descriptor for universal multimedia 
access application relates to fidelity of a summary abstraction of a 
program. This preference descriptor may correspond to the variation 
fidelity descriptor contained in the media's variation description 
specified by Variation Description Scheme in ISO/IEC JTC1/SC29 
WGl 1 N3246, "MPEG-7 Multimedia Description Schemes, Working 
Draft v2.0", Noordwijkerhout, March 2000. Alternatively, the duration 
and number of keyfirames may be defined as the fidelity descriptor. 

(D) The SegmentTheme descriptor(s) may be included, w^hich 
describes the theme or point of view of a segment, e.g., a video or 
audio clip annotated with its theme or emphasis point. An example 
summary preference description expressing preference for video 
segments (clips) labeled as "Goal fi*om Spain" and "Replay of Goal 
fi-om Spain" is as follows: 



</UsagePreferences> 
</BrowsingPreferences> 
<SummaryPreferences> 

<SummaryTypePreference>KeyVideoClips<ySiiimnaryTypePreference> 
<SegmentTheme>Goal from SpaiiK/SegmentThemO 
<SegmentTheme>Replay of goal from Spain<;/SegmentTheme> 
</SummaryPreferences> 
</BrowsmgPreferences> 
</UsagePreferences> 

(E) The fi*ame fi-equency value descriptor may be included to 

specify the temporal sampling frequency of video frames that can be 
visualized in the browser. The frames provide a visual summary. 
Depending on the browser, they may also provide clickable entry 
points to the video. The user may click and start playing back the 
video starting from that frame. The frame frequency value descriptor 
provides similar functionality in terms of shots of the video. 
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The source preference description describes the preferred source of 
muhimedia information, such as the broadcast or storage medium type (e.g., 
terrestrial, sateUite, DVD), broadcast channel identifier, etc. An example user 
preference description expressing preference for Star Trek episodes available from 
5 terrestrial broadcast is as follows. 

<UserIdentifler protection="tnie" userName="paurV> 
<UsagePreferences alio wAutomaticUpdate=" false "> 
<FilteringAndSearchPreferences protection="true"> 
<PreferenceValue>5</PreferenceValue> 
1 0 <CreationPreferences> 

<Title xnil:lang="en" type="original">Star Trek</Title> 
Q </CreationPreferences> 

lG 

5 ~ <SourcePreferences> 

M <PublicationType>Terrestrial Broadcast<yPublicationType> 

Q 

15 </SourcePreferences> 

Q </FilteringAndSearchPreferences> 

^ <AJ sagePreferences> 

Q <yUserIdentifier> 

i ri 

=5 I 

01 2 0 The filtering and search preferences description includes at least one of 

S the descriptors of preferred program title, genre, language, actor, creator of the 

5 — s 

program. An example description where user's preference is for news programs in 
English is given below. Such description may be included in user's smart card when 
he travels to Japan, for example. Note that this particular preference description is 

2 5 identified as being specific to Japan and differentiated by choosing an appropriate user 

name. 

<UserIdentifier protection="true" userName="paul_in_Japan"/> 
<UsagePreferences allowAutomaticUpdate^" false "> 
<FilteringAndSearchPreferences protection="true"> 

3 0 <Preference Value> 1 00</Preference Value> 

<ClassificationPreferences> 
<Language> 

<LanguageCode>en</LanguageCode> 
</Language> 
3 5 <Genre>News</Genre> 



</ClassificationPreferences> 
</FilteringAndSearchPreferences> 
</UsagePreferences> 
</UserIdentifier> 



The Preference Value descriptor provides a technique for prioritizing 
filtering and search preferences, such as the value indicating the degree of user's 
preference or non-preference. Non-preferences may be expressed by assigning a 
negative (opposite) value to the preference value descriptor. 

The betterthan and worsethan descriptors may describe which 
instantiation of preferences the user likes or dislikes relatively more compared to 
another instantiation, where different instantiations are identified using the filtering 
and search preference type descriptor. This provides robustness against changes in the 
preference value descriptor automatically, for example, by an agent. 

The filtering and search preferences description may also contain a 
description of a preferred review to express user's desire for searching for programs 
that are favorably reviewed by specific individuals. For example, preference for 
movies reviewed by movie critics Siskel and Ebert and found to be "two-thumbs- up" 
may be described and included in the filtering and search preferences description. 

An overview of the entire description scheme is shown in FIG. 29. 

Additional Attributes and Descriptors 

The present inventors came to the additional realization that enhanced 
functionality for the system may be achieved by the incorporation of particular types 
of information in the descriptions and description schemes. In addition, these 
particular types of information may likewise be incorporated within systems without 
descriptions and description schemes to provide enhanced fimctionality. An 
explanation of the additional attributes and descriptors is hereinafter provided, 
followed by a portion of example descriptions. 

After fiirther consideration, in particular with respect to the user 
interaction with the system, there is a need for the user preference descriptions to 
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include descriptions of the user's preference that relate to how media is recorded in the 
local storage of the system. The media may include, for example, audio and/or video 
content. For instance, the user may want to express a preference for the quality of the 
recording technique. The quality may include, for example, low quality, medium 
quality, and high quality. These quality levels refer to the quality of the audio and/or 
video compression technique that is applied to the media prior to being stored on the 
local storage. Preferably, the audio and/or video content is digital and a digital 
compression technique is employed. The preference of the recording quality may be 
influenced by the available capacity of the local storage or attributes of the program. 
Accordingly, the system may suggest, select different recording qualities, and/or 
otherwise record programs at different recording qualities based upon the available 
local storage. 

Current personal recording devices, such as the Replay TV personal 
video recorder, include a fixed single default recording quality that can be chosen by 
the user by invoking the setup mode. The default setting may be for standard, 
medium or high quality. The user manually has to change this default setting to one of 
the other two options, if desired, whenever the user instructs the Replay TV personal 
recording device to record an upcoming program. In addition, the Replay TV 
personal recording device records every show that the user is watching in order to 
support live pause and replay functionality. However, the present inventors were 
surprised to observe that the recording quality of the Replay TV personal recording 
device during watching is fixed and cannot be adjusted. 

Including the recording quality preferences in the user preference 
descriptions facilitates certain, previously unrealized, advantages in a flexible and 
efficient manner. By using the recording quality preference descriptions in 
connection with the other aspects of the usage preference descriptions, for example 
with the filtering and search preference descriptions, recording preferences may be 
automatically associated with a particular type of content. For example, the system 
may automatically set to recording quality to "high" for sports programming and 
"low" for nature programming. Sports programming tends to include a lot of fast 
action which is preferably encoded with high quality. In contrast, nature 



programming tends to primarily include little action, which if encoded with low 
quality, will still look acceptable while saving on the storage requirements of the 
system. 

The system may select the recording quality of programs (audio and/or 
video) based on predefined rules for particular attributes included within the system or 
associated with the audio and/or video content. In this manner, the system may 
accommodate for differences in the anticipated content of audio and/or video 
programs. Thus, the user does not have to make manual choices every time when the 
user desires to record different programs at different qualities. The system may have 
access to information regarding both the user's preferred content and the preferred 
recording qualities for such content, enabling the system to potentially function in an 
autonomous manner on behalf of the user. In this manner, the system may discover 
and record preferred programs at the user*s desired recording qualities. Accordingly, 
the system may permit the user to select the recording quality of programs (audio 
and/or video) based on predefined rules for particular attributes included within the 
system or associated with the audio and/or video content. 

The system may include a learning agent that automatically observes 
and learns the user's recording preferences for particular types of content. If desired, 
the user may input his preferences into his profile. User preferences facilitate multi- 
user support for personalized recording quality settings for all the users of the device. 
Also, this likewise permits manual and/or automatic selection of the recording quality 
for recording by the system while viewing. In this manner, the recording quality of 
the program being currently watched may be selected based on the content of the 
program or the user's preferences. It is to be understood that the system may include 
manual and/or automatic selection of the recording quality in a system that does not 
include descriptions and description schemes. 

The user may desire to express their preferences of safeguard time 
intervals that precede and follow the program content in an effort to account for 
possible shifts in program start and end times. For example, a program may be 
scheduled to be broadcast between 10am and 1 lam on a particular channel, where the 
user selects a safeguard time interval of 5 minutes before the program starts and a 
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safeguard time interval of 10 minutes after the program ends, resulting in a recording 
time from 9:55am to 1 1 : 10am. These safeguard time intervals may be based upon the 
content of the user descriptions and description schemes. This is particularly usefiil in 
programming the system in advance to record one or more programs without the 
5 user's explicit intervention to set the safeguard time intervals, which is burdensome to 
many users. For example, because sports programming typically extends beyond the 
scheduled time the system may include significant additional safeguard time interval 
at the end of such programming while sitcoms typically end when scheduled so little, 
if any, additional safeguard time interval is normally necessary. It is to be understood 
1 0 that the system may include manual and/or automatic selection of the safeguard time 
O intervals in a system that does not include descriptions and description schemes. 

|n The present inventors also came to the realization that consideration 

f=i should likewise be given to recording of audio and/or visual content that is enhanced 

with hyperlinks, e.g., multiple layers of links to web pages that are related to the 
M 15 content, for subsequent, off-line consumption. In this case, the user records not only 

p the audiovisual program but also its enhancement data as indicated by the hyperlinks. 

~ ! It is desirable for the user to be able to specify the maximum number of layers of links 

5 that the device should automatically record in order to limit the necessary storage 

□ requirements in some manner. In addition, a parameter may be selected to indicate 

2 0 the maximum amount of local storage to use for such enhancement data which reduce 
the storage requirements and the time necessary for downloading the enhancement 
data. Selecting the number of layers of links to store assists the user in rationing the 
limited locate storage space in the system. By including the number of link layers as 
part of the user preferences, it is possible for the users to automatically associate such 

2 5 preferences with media characteristics, such as the genre or any other attribute 

associated with the program. For example, a large number of link layers may be 
specified for educational interactive programs. In addition, the multi-user support 
provided by the user preference description provides personalized specifications for 
all users of the appliance. It is to be understood that the system may include manual 

3 0 and/or automatic selection of the link level or maximum storage permitted in a system 

that does not include descriptions and description schemes. 
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As previously described, the system may include device preferences 
that describe user preferences relating to how the user consmnes the media using the 
device. Consuming the media includes utilization of the so-called "trick modes" of 
the device, namely, fast forward and fast reverse. Fast forward and fast reverse 
include any speed faster than normal playback speed. Digital random access storage 
media based appliances facilitate a wide range of fast forward and fast reverse speeds 
(e.g., 4 times versus 20 times the normal speed) as well as instant skipping by a 
particular amount (e.g., 15 seconds) from one point to another point in the content. 
Skipping, when' allowed, may be used to skip commercials or uninteresting parts of 
programs. After consideration of the capability of fast forwarding, fast reverse, and 
skipping, the present inventors came to the realization that a need exists to provide a 
technique so that the user does not have to select among these multiple parameters 
every time he or she invokes these device fimctionalities. The Browsing Preference 
descriptions, previously described, may include the playback preferences that in turn 
include the fast forward, fast reverse, and instant skip preference descriptions. By 
making such preference descriptions a part of the user preference descriptions, users 
are provided with the opportunity to customize such preferences to content 
characteristics such as the genre or any other attribute associated with a program. In 
this way, the user does not have to make choices each time the function is used. The 
user may select his or her preferences for different programming differentiated on any 
basis. Altematively, an agent may automatically leam and populate the user's user 
preference descriptions to provide personalized settings for all the users of the 
appliance. Furthermore, users may edit their user preferences thereby changing their 
preference on the instant skip duration. Likewise, this technique may be similarly 
used for slow forward and slow reverse, namely, a speed slower than normal playback 
speed. 

Another set of useful additional information is related to filtering and 
search preferences, particularly, to preferences on the format of the preferred content. 
After consideration the present inventors came to the conclusion that there is a need 
for describing the user's preference for a particular media format. This is especially 
desirable considering that the content may be available in multiple formats, or when 
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the user's device has limited playback capabilities of various media fomiats. The 
source preferences may include a preferred file format description. Examples of some 
media formats may include, for example, MPEG-1, MPEG-2, MP3, Liquid Audio, 
Real Player, AC-3, Dolby Digital, Wide Screen video format, and Normal Screen 
video format. It is to be imderstood that the media format maybe used in a system that 
does not include descriptions and description schemes. 

While the system described herein includes extensive search, 
characterization, browsing, and filtering fiinctions the present inventors came to the 
realization that yet another suitable program attribute is the date of the creation of the 
content itself For example, the creation date for a particular movie may be the date 
during which the original movie was created, such as 1930, 1940-1949, or 2000. 
Also, the creation date for a particular movie may be the date on which the movie was 
re-mastered into its current format, such as DVD. With the creation date data, movies 
created by a particular director between 1960 and 1970, or episodes of Star Trek that 
were created between 1977 and 1982 may be selected. The creation date is 
particularly useful in the filtering out of reruns of the user's favorite television 
episodes. Otherwise, it may be difficult for the system to distinguish between new (or 
relatively new) television episodes and old (or relatively old) television episodes. The 
date period may support any desired granularity, such as years, decades, hours, 
months, etc. 

Expression of the Additional Attributes 
The following descriptions are expressed in XML (Extensible Markup 
Language), incorporated by reference herein. It is to be imderstood that any other 

2 5 description language may likewise be used. 

The definition of the user preference description may be as follows. In 
the following the user Paul decides to define a user preference description specifically 
for sports. Paul's recording quality preference for this particular genre is "High". Paul 
also specifies a longer safeguard period at the end of a sports program (15 minutes) 

3 0 than the safeguard at the start (30 seconds), due to possible extensions in the game. 

Paul expresses preference for single layer of -linked content. 
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<UserPreference> 

<UserIdentifier protection="true" userName="pPaul_sports"/> 
<UsagePreferences allow AutomaticUpdate="false"> 
<FilteringAndSearchPreferences protection="true*'> 
<ClassificatiorLPreferences> 
<Genre>Sports</Genre> 
</ClassificationPreferences> 
</F ilteringAndSearchPreferences> 
<DevicePreferences> 

<RecordingPreferences> 

<Quality>High</Quality> 
<SafeguardStartOffset> 

<s>30</s> 
</SafeguardStartOffset> 
<SafeguardEndOffset> 

<m>15</m> 
</SafeguardEndOffset> 
<LinkLayers> 1 </LinkLayers> 
</RecordingPreferences> 
</DevicePreferences> 
</UsagePreferences> 
<AJserIdentifier> 
</UserPreference> 

In the above example description, recording preferences are included in 
device preferences. In another alternative embodiment, recording preferences may be 
included in filtering and search preferences. In this manner, recording preferences can 
be associated with each individual set of filtering and search preferences. It is then 
possible to automatically associate different recording preferences with different 
preferred title, genre, or any other description of preferred content, or their 
combinations. The inclusion can be explicit, or implicit by making use of links to a 
separate recording preference which may in turn be included in the device preferences. 
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In this case, filtering and search preferences may also include a binary valued 
attribute to indicate user's preference for to record or not to record those programs that 
meet user's filtering and search preferences. Recording preferences take effect if 
user's preference is to record. 
5 Further, additional program-specific recording related preferences may 

be included in the filtering and search preferences, if the particular instantiation of the 
filtering and search preferences sufficiently identifies a particular program. In that 
case, for example, if the preferred program has multiple, regularly scheduled episodes, 
the following preferences may be specified: the number of episodes that is desired to 
10 be recorded (e.g., 1,2, All), and the number of most recent episodes that is desired 
5 to be kept in local storage (e.g., if this nimiber is 1, a new episode always replaces the 

m previous episode.), and whether the system should immediately reserve space for this 

p particular program if it is going to be available in the fixture. 

2 Playback preferences on fast forward and fast reverse rates, and instant 

K 15 skip amount may be as follows. 

□ <UserPreference> 

^ \ <UserIdentifier protection="true" userName="paurV> 

<UsagePreferences allow AutomaticUpdate='Talse"> 
Q <BrowsingPreferences> 

2 0 ... ' 

<PlaybackPreferences> 

<FfwdSpeed>5</FFwdSpeed> 
<FrevSpeed>3</FRevSpeed> 
<InstantSkip><s>l 5</s></InstantSkip> 
2 5 </PlaybackPreferences> 



</BrowsingPreferences> 

<FilteringAndSearchPreferences protection="true"> 



30 



85 

</FilteringAndSearchPreferences> 
<DevicePreferences> 



<RecordingPreferences> 

<Quality>Medium</Quality> 
<SafeguardStartOffset> 

<m>l</m> 
</SafeguardBegin> 
<S afeguardEnd> 

<m>l/m> 
</SafeguardEndOffset> 
<LinkLay ers>3 </LinkLayers> 
</RecordingPreferences> 

</DevicePreferences> 
</UsagePreferences> 
</UserIdentifier> 
</UserPreference> 

In the following description, the user specifies preference for MPEG-1 

file format. 
<UserPreference> 

<UserIdentifier protection="true" userName="paur'/> 

<UsagePreferences allow AutomaticUpdate="false"> 

<FilteringAndSearchPreferences protection="true"> 



</CreationPreferences> 
<SourcePreferences> 

<PubhcationType>Terrestrial Broadcast</PublicationType> 
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</SourcePreferences> 

<MediaFormatPrefereiices> 

<FileFormat>MPEG- 1 </FileFormat> 
</MediaF ormatPreferences> 



</FilteringAndSearchPreferences> 

</UsagePreferences> 
1 0 </UserIdentifier> 
D </UserPreference> 

SSL 

yj 

y I 

^ The following description includes user's preference on creation date period as a part 

S of creation preferences. The user has a preference for Star Trek episodes created 

m 15 between 1977 and 1982. In this example, the user has preference on the source as 

terrestrial broadcast. 
<UserPreference> 

<UserIdentifier protection="true" userNanie="paurV> 
<UsagePreferences allowAutomaticUpdate=" false"> 
2 0 <FilteringAndSearchPreferences protection="true"> 

<CreationPreferences> 

<Title xml:lang="en" type="originar'>Star Trek</Title> 
<DatePeriod> 

<TimePoint> 

2 5 <y>1977</y> 

</TimePoint> 
<Duration> 

<y>5</y> 
</Duration> 

3 0 </DatePeriod> 

</CreationPreferences> 
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<SourcePreferences> 

<PublicationType>Terrestrial Broadcast</PublicationType> 
</SourcePreferences> 
</FilteringAiidSearchPreferences> 
5 <AJsagePreferences> 
</UserIdentifier> 
</UserPreference> 

It is to be understood that any of the descriptors and description 
schemes may be located at any portion of the system. Merely for matters of 
10 illustration, referring to FIG. 30 a recording preferences may include the recording 
Q quality, safeguard time intervals, and number of linked layers. A playback 

m preferences may include the fast forward, fast reverse, and instant skip period. A 

media format may include the preferred media format and a creation date may include 
^ the preferred creation date penod. 

W 15 The terms and expressions that have been employed in the foregoing 

s 

Q specification are sued as terms of description and not of limitation, and there is no 

^ I intention, in the use of such terms and expressions, of excluding equivalents of the 

^ features show^n and described or portions thereof, it being recognized that the scope of 

o 

Q the invention is defined and limited only by the claims that follow^. 
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